[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:45:49 dist-test-slave-9tm1 krb5kdc[2380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:45:49 dist-test-slave-9tm1 krb5kdc[2380](info): set up 2 sockets
Sep 24 12:45:49 dist-test-slave-9tm1 krb5kdc[2380](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 12:45:50.782392 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:45:50 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:45:50.953881 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39891
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:51.284403 2393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:51.284734 2393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:51.284857 2393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:51.292657 2393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:45:51.292809 2393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:51.292882 2393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:51.292956 2393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:45:51.293049 2393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:45:51.303488 2393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:51.304877 2393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:51.306313 2393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:51.319378 2401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:51.319603 2398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:51.319650 2399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:51.320699 2393 server_base.cc:1034] running on GCE node
I20260924 12:45:51.321640 2393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:51.323158 2393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:51.324383 2393 hybrid_clock.cc:648] HybridClock initialized: now 1790253951324297 us; error 69 us; skew 500 ppm
I20260924 12:45:51.324743 2393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:51 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253951, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:51.327981 2393 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:45:51.330008 2393 webserver.cc:460] Webserver started at http://127.2.81.126:43951/ using document root <none> and password file <none>
I20260924 12:45:51.330705 2393 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:51.330806 2393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:51.331061 2393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:51.334270 2393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763"
format_stamp: "Formatted at 2026-09-24 12:45:51 on dist-test-slave-9tm1"
server_key: "a8d9aa5882e01829c51c11e2f9a8ca90"
server_key_iv: "4e4c90bda6e925334e7f04013af47cfd"
server_key_version: "encryptionkey@0"
I20260924 12:45:51.335209 2393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763"
format_stamp: "Formatted at 2026-09-24 12:45:51 on dist-test-slave-9tm1"
server_key: "a8d9aa5882e01829c51c11e2f9a8ca90"
server_key_iv: "4e4c90bda6e925334e7f04013af47cfd"
server_key_version: "encryptionkey@0"
I20260924 12:45:51.341245 2393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:45:51.345252 2408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:51.346939 2393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:45:51.347147 2393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763"
format_stamp: "Formatted at 2026-09-24 12:45:51 on dist-test-slave-9tm1"
server_key: "a8d9aa5882e01829c51c11e2f9a8ca90"
server_key_iv: "4e4c90bda6e925334e7f04013af47cfd"
server_key_version: "encryptionkey@0"
I20260924 12:45:51.347362 2393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:51.363808 2393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:51.368403 2393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:51.368696 2393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:51.388046 2393 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:39891
I20260924 12:45:51.388074 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:39891 every 8 connection(s)
I20260924 12:45:51.389619 2393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:45:51.393707 2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:51.400430 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 2393
I20260924 12:45:51.400703 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:45:51.402657 2373 external_mini_cluster.cc:1428] Setting key 82f38072a8ca3203ef363bc8d382e0ba
I20260924 12:45:51.404337 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763: Bootstrap starting.
I20260924 12:45:51.407912 2460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:51.409365 2460 log.cc:824] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:51.412209 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763: No bootstrap required, opened a new log
I20260924 12:45:51.421731 2460 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39891 } }
I20260924 12:45:51.422251 2460 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:51.422415 2460 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce60e16af7ee42ff9eb4b3bf7dbca763, State: Initialized, Role: FOLLOWER
I20260924 12:45:51.423079 2460 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39891 } }
I20260924 12:45:51.423338 2460 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:45:51.423468 2460 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:45:51.423641 2460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:51.425909 2460 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39891 } }
I20260924 12:45:51.426476 2460 leader_election.cc:302] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce60e16af7ee42ff9eb4b3bf7dbca763; no voters:
I20260924 12:45:51.427139 2460 leader_election.cc:288] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:45:51.427819 2465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:45:51.429067 2465 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [term 1 LEADER]: Becoming Leader. State: Replica: ce60e16af7ee42ff9eb4b3bf7dbca763, State: Running, Role: LEADER
I20260924 12:45:51.429814 2465 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39891 } }
I20260924 12:45:51.430613 2460 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:45:51.433470 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce60e16af7ee42ff9eb4b3bf7dbca763. Latest consensus state: current_term: 1 leader_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39891 } } }
I20260924 12:45:51.433871 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:51.434437 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce60e16af7ee42ff9eb4b3bf7dbca763" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39891 } } }
I20260924 12:45:51.434723 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763 [sys.catalog]: This master's current role is: LEADER
Sep 24 12:45:51 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:51.447419 2471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:45:51.456058 2471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:45:51.461882 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:51.407569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.2.81.126:39891)
0924 12:45:51.408682 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:51.408700 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:51.410399 (+ 1699us) server_negotiation.cc:407] Connection header received
0924 12:45:51.421069 (+ 10670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:51.421108 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:51.421611 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:51.422060 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:51.427026 (+ 4966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:51.430110 (+ 3084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:51.432494 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:51.433820 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:51.434094 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:51.443973 (+ 9879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:51.443988 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:51.444015 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:51.444082 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:51.450064 (+ 5982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:51.451590 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:51.451603 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:51.451614 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:51.451903 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:51.453341 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:51.453349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:51.453353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:51.455048 (+ 1695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:51.455667 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:51.458006 (+ 2339us) server_negotiation.cc:299] Negotiation successful
0924 12:45:51.458619 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":585,"threads_started":1}
I20260924 12:45:51.471127 2471 catalog_manager.cc:1347] Generated new cluster ID: ca43136b9fd5474b8396b583000181a1
I20260924 12:45:51.471478 2471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:45:51.491367 2471 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:45:51.492883 2471 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:45:51.502362 2471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce60e16af7ee42ff9eb4b3bf7dbca763: Generated new TSK 0
I20260924 12:45:51.503597 2471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:45:51.573710 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39891
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:51.952577 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:51.953447 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:51.953634 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:51.963591 2486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:51.963766 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:51.963912 2486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:45:51.974710 2486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39891
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:51.976011 2486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:51.977439 2486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:51.992447 2491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:51.992460 2494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:51.992532 2492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:51.993268 2486 server_base.cc:1034] running on GCE node
I20260924 12:45:51.993906 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:51.994904 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:51.996145 2486 hybrid_clock.cc:648] HybridClock initialized: now 1790253951996071 us; error 65 us; skew 500 ppm
I20260924 12:45:51.996491 2486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:51 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253951, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:51.999791 2486 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:45:52.001623 2486 webserver.cc:460] Webserver started at http://127.2.81.65:42901/ using document root <none> and password file <none>
I20260924 12:45:52.002532 2486 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:52.002677 2486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:52.003013 2486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:52.006729 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "088f1c473bf549e3ad078a42f7150c6f"
format_stamp: "Formatted at 2026-09-24 12:45:51 on dist-test-slave-9tm1"
server_key: "32c8094f6252607c0101392ddb35b511"
server_key_iv: "12354cfc71b811c971726c5302099962"
server_key_version: "encryptionkey@0"
I20260924 12:45:52.007596 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "088f1c473bf549e3ad078a42f7150c6f"
format_stamp: "Formatted at 2026-09-24 12:45:51 on dist-test-slave-9tm1"
server_key: "32c8094f6252607c0101392ddb35b511"
server_key_iv: "12354cfc71b811c971726c5302099962"
server_key_version: "encryptionkey@0"
I20260924 12:45:52.013464 2486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 12:45:52.017349 2501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:52.018961 2486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:45:52.019156 2486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "088f1c473bf549e3ad078a42f7150c6f"
format_stamp: "Formatted at 2026-09-24 12:45:51 on dist-test-slave-9tm1"
server_key: "32c8094f6252607c0101392ddb35b511"
server_key_iv: "12354cfc71b811c971726c5302099962"
server_key_version: "encryptionkey@0"
I20260924 12:45:52.019358 2486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:52.037842 2486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:52.042604 2486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:52.042922 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:52.044523 2486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:52.046054 2486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:52.046139 2486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:52.046262 2486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:52.046321 2486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:52.082454 2486 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:38147
I20260924 12:45:52.082512 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:38147 every 8 connection(s)
I20260924 12:45:52.083814 2486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:45:52.094274 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 2486
I20260924 12:45:52.094532 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:45:52.095269 2373 external_mini_cluster.cc:1428] Setting key 18e2236548784a562b2b1307f11f9f3b
Sep 24 12:45:52 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253951, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:52.108784 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:52.086930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:41251 (local address 127.2.81.126:39891)
0924 12:45:52.087406 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:52.087410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:52.088721 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 12:45:52.090738 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:52.090744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:52.090836 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:52.090934 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:52.092654 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.093436 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.095983 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.096543 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.096692 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:52.101431 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:52.101437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:52.101440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:52.101480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:52.105206 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:52.106177 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:52.106183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:52.106186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:52.106343 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:52.106762 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:52.106767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:52.106770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:52.107116 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:52.107268 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:52.108149 (+ 881us) server_negotiation.cc:299] Negotiation successful
0924 12:45:52.108409 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260924 12:45:52.109835 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:52.087483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39891 (local address 127.2.81.65:41251)
0924 12:45:52.088509 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:52.088564 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:52.090408 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:52.091230 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:52.091240 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:52.091821 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:45:52.092445 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:52.092466 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.093603 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.093606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:52.095736 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:52.095757 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.096832 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.096838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:52.097122 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:52.097891 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:45:52.097916 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:45:52.101070 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:45:52.105471 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:52.105481 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:52.105497 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:52.106020 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:52.106454 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:52.106457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:52.106460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:52.106657 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:52.107312 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:45:52.107319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:45:52.107708 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 12:45:52.108021 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 12:45:52.108485 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:45:52.111865 2614 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39891
I20260924 12:45:52.112193 2614 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:52.113080 2614 heartbeater.cc:511] Master 127.2.81.126:39891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:45:52.115710 2425 ts_manager.cc:194] Registered new tserver with Master: 088f1c473bf549e3ad078a42f7150c6f (127.2.81.65:38147)
I20260924 12:45:52.117343 2425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:41251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:45:52.138067 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39891
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:52.487097 2620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:52.487411 2620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:52.487552 2620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:52.494918 2620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:52.495035 2620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:52.495183 2620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:45:52.505676 2620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39891
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:52.506994 2620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:52.508281 2620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:52.521090 2625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:52.520931 2628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:52.521590 2626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:52.521795 2620 server_base.cc:1034] running on GCE node
I20260924 12:45:52.522435 2620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:52.523470 2620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:52.524688 2620 hybrid_clock.cc:648] HybridClock initialized: now 1790253952524631 us; error 56 us; skew 500 ppm
I20260924 12:45:52.525031 2620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:52 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253952, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:52.528384 2620 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:45:52.529834 2620 webserver.cc:460] Webserver started at http://127.2.81.66:41613/ using document root <none> and password file <none>
I20260924 12:45:52.530537 2620 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:52.530676 2620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:52.530970 2620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:52.534035 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "500d640114ac4cb08a5f5c903e70033e"
format_stamp: "Formatted at 2026-09-24 12:45:52 on dist-test-slave-9tm1"
server_key: "24ba972c90bd699f2a2181e67f7bd5a9"
server_key_iv: "adbcb717e2dc9c042f37067900d254ca"
server_key_version: "encryptionkey@0"
I20260924 12:45:52.534755 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "500d640114ac4cb08a5f5c903e70033e"
format_stamp: "Formatted at 2026-09-24 12:45:52 on dist-test-slave-9tm1"
server_key: "24ba972c90bd699f2a2181e67f7bd5a9"
server_key_iv: "adbcb717e2dc9c042f37067900d254ca"
server_key_version: "encryptionkey@0"
I20260924 12:45:52.539827 2620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:45:52.543481 2635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:52.545149 2620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:45:52.545323 2620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "500d640114ac4cb08a5f5c903e70033e"
format_stamp: "Formatted at 2026-09-24 12:45:52 on dist-test-slave-9tm1"
server_key: "24ba972c90bd699f2a2181e67f7bd5a9"
server_key_iv: "adbcb717e2dc9c042f37067900d254ca"
server_key_version: "encryptionkey@0"
I20260924 12:45:52.545527 2620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:52.560503 2620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:52.564872 2620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:52.565192 2620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:52.566182 2620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:52.567725 2620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:52.567827 2620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:52.567953 2620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:52.568010 2620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:52.605677 2620 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:43207
I20260924 12:45:52.605736 2747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:43207 every 8 connection(s)
I20260924 12:45:52.606931 2620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:45:52.615621 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 2620
I20260924 12:45:52.615847 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:45:52.616247 2373 external_mini_cluster.cc:1428] Setting key 0e90bd06ba9743b5000babcc5551ff83
Sep 24 12:45:52 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253952, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:52.629632 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:52.609926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:41841 (local address 127.2.81.126:39891)
0924 12:45:52.610334 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:52.610340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:52.611849 (+ 1509us) server_negotiation.cc:407] Connection header received
0924 12:45:52.613744 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:52.613749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:52.613826 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:52.613934 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:52.615748 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.616525 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.618473 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.618904 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.619009 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:52.623521 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:52.623527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:52.623531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:52.623567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:52.626135 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:52.627074 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:52.627079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:52.627081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:52.627241 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:52.627761 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:52.627766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:52.627768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:52.628061 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:52.628196 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:52.629008 (+ 812us) server_negotiation.cc:299] Negotiation successful
0924 12:45:52.629247 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":184,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:45:52.630645 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:52.610468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39891 (local address 127.2.81.66:41841)
0924 12:45:52.611221 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:52.611265 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:52.613455 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:52.614198 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:52.614208 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:52.614801 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:45:52.615519 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:52.615548 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.616695 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.616700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:52.618303 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:52.618321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:52.619128 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:52.619134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:52.619452 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:52.620201 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:45:52.620241 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:45:52.623222 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:45:52.626324 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:52.626336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:52.626357 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:52.626863 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:52.627389 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:52.627394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:52.627398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:52.627626 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:52.628195 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:45:52.628205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:45:52.628602 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 12:45:52.628844 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 12:45:52.629456 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":299,"threads_started":1}
I20260924 12:45:52.632747 2748 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39891
I20260924 12:45:52.633131 2748 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:45:52.633926 2748 heartbeater.cc:511] Master 127.2.81.126:39891 requested a full tablet report, sending...
I20260924 12:45:52.635712 2425 ts_manager.cc:194] Registered new tserver with Master: 500d640114ac4cb08a5f5c903e70033e (127.2.81.66:43207)
I20260924 12:45:52.636739 2425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:41841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:45:52.657712 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39891
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:53.011925 2754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:53.012238 2754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:53.012336 2754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:53.019626 2754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:53.019759 2754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:53.019878 2754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:45:53.030438 2754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39891
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:53.031746 2754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:53.033190 2754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:53.048367 2759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:53.048291 2760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:53.048288 2762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:53.051452 2754 server_base.cc:1034] running on GCE node
I20260924 12:45:53.052155 2754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:53.053804 2754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:53.055305 2754 hybrid_clock.cc:648] HybridClock initialized: now 1790253953055247 us; error 66 us; skew 500 ppm
I20260924 12:45:53.055661 2754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:53.059665 2754 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:45:53.061599 2754 webserver.cc:460] Webserver started at http://127.2.81.67:37837/ using document root <none> and password file <none>
I20260924 12:45:53.062376 2754 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:53.062503 2754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:53.062803 2754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:53.066103 2754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "3bf718dfe3774294b9a2f768dd9e7c22"
format_stamp: "Formatted at 2026-09-24 12:45:53 on dist-test-slave-9tm1"
server_key: "980000a291929f2681307d2bcef23ad5"
server_key_iv: "d919de7253b9e299de3ab1bc6cedc9ae"
server_key_version: "encryptionkey@0"
I20260924 12:45:53.066941 2754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "3bf718dfe3774294b9a2f768dd9e7c22"
format_stamp: "Formatted at 2026-09-24 12:45:53 on dist-test-slave-9tm1"
server_key: "980000a291929f2681307d2bcef23ad5"
server_key_iv: "d919de7253b9e299de3ab1bc6cedc9ae"
server_key_version: "encryptionkey@0"
I20260924 12:45:53.072456 2754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:45:53.076516 2769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:53.078200 2754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:45:53.078406 2754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "3bf718dfe3774294b9a2f768dd9e7c22"
format_stamp: "Formatted at 2026-09-24 12:45:53 on dist-test-slave-9tm1"
server_key: "980000a291929f2681307d2bcef23ad5"
server_key_iv: "d919de7253b9e299de3ab1bc6cedc9ae"
server_key_version: "encryptionkey@0"
I20260924 12:45:53.078620 2754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:53.110986 2754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:53.115669 2754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:53.115978 2754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:53.117121 2754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:53.119146 2754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:53.119228 2754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:53.119345 2754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:53.119407 2754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:53.121274 2614 heartbeater.cc:503] Master 127.2.81.126:39891 was elected leader, sending a full tablet report...
I20260924 12:45:53.153133 2754 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:37097
I20260924 12:45:53.153184 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:37097 every 8 connection(s)
I20260924 12:45:53.155148 2754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:45:53.155546 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 2754
I20260924 12:45:53.155747 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:45:53.156277 2373 external_mini_cluster.cc:1428] Setting key b22a2a88bbb8b50cab1a5701e4d810ff
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:53.177049 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.157880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:52271 (local address 127.2.81.126:39891)
0924 12:45:53.161384 (+ 3504us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:53.161390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:53.161407 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:45:53.161985 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:53.161990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:53.162098 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:53.162208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:53.163912 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.164655 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.166299 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.166674 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.166796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.170474 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:53.170479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:53.170482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:53.170518 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:53.173538 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.174387 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.174391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.174394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.174526 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.175043 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.175047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.175049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.175331 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:53.175474 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:53.176274 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 12:45:53.176509 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3381,"thread_start_us":3265,"threads_started":1}
I20260924 12:45:53.178069 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.158498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39891 (local address 127.2.81.67:52271)
0924 12:45:53.159334 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:53.159387 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:53.161710 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:53.162475 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:53.162485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:53.163125 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:45:53.163730 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:53.163750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.164844 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.164848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:53.166150 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:53.166162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.166890 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.166897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:53.167125 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.167787 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:45:53.167810 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:45:53.170216 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:45:53.173708 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:53.173716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:53.173731 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:53.174203 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:53.174638 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:53.174643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:53.174647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:53.174927 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:53.175456 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:45:53.175464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:45:53.175850 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 12:45:53.176144 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 12:45:53.176783 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":329,"threads_started":1}
I20260924 12:45:53.179610 2882 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39891
I20260924 12:45:53.179904 2882 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:53.180748 2882 heartbeater.cc:511] Master 127.2.81.126:39891 requested a full tablet report, sending...
I20260924 12:45:53.182694 2425 ts_manager.cc:194] Registered new tserver with Master: 3bf718dfe3774294b9a2f768dd9e7c22 (127.2.81.67:37097)
I20260924 12:45:53.183913 2425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:52271
I20260924 12:45:53.188723 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:45:53.234894 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.218625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.2.81.65:38147)
0924 12:45:53.219272 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:53.219280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:53.219307 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:45:53.219421 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:53.219428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:53.219644 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:53.219774 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:53.221438 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.222470 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.223925 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.224745 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.224866 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.229237 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:53.229252 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:53.229269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:53.229315 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:53.232109 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.232839 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.232846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.232848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.232994 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.233391 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.233396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.233400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.233735 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:53.234027 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:53.234299 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 12:45:53.234540 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":371,"threads_started":1}
W20260924 12:45:53.236099 2529 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52524
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:45:53.279356 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.265152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.2.81.65:38147)
0924 12:45:53.265363 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:53.265369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:53.265579 (+ 210us) server_negotiation.cc:407] Connection header received
0924 12:45:53.265752 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:53.265757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:53.265828 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:53.265913 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:53.267204 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.267868 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.269530 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.269962 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.270073 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.273993 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:53.273998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:53.274002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:53.274042 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:53.276751 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.277582 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.277587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.277589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.277725 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.278126 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.278129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.278131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.278419 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:53.278594 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:53.278918 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 12:45:53.279086 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:45:53.281103 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 2486
I20260924 12:45:53.293565 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.281983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.2.81.65:38147)
0924 12:45:53.282142 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:53.282148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:53.282542 (+ 394us) server_negotiation.cc:407] Connection header received
0924 12:45:53.282692 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:53.282698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:53.282762 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:53.282849 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:53.284101 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.284918 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.286176 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.286615 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.286711 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.288859 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:53.288864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:53.288867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:53.288904 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:53.291281 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.291967 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.291971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.291973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.292100 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.292376 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.292379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.292381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.292620 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:53.292790 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:53.293146 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 12:45:53.293297 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 12:45:53.294523 2609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:53.378453 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 2486
I20260924 12:45:53.391728 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 2620
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:45:53.407843 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.392704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.2.81.66:43207)
0924 12:45:53.393293 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:53.393302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:53.393334 (+ 32us) server_negotiation.cc:407] Connection header received
0924 12:45:53.393403 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:53.393412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:53.393688 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:53.393845 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:53.395004 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.396051 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.397492 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.398059 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.398165 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.401984 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:53.401997 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:53.402012 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:53.402051 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:53.404966 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.405760 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.405768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.405771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.405914 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.406357 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.406361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.406363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.406775 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:53.407017 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:53.407354 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 12:45:53.407562 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":289,"threads_started":1}
I20260924 12:45:53.409132 2743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:53.489996 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 2620
I20260924 12:45:53.507483 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 2754
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:45:53.523478 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:53.508453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.2.81.67:37097)
0924 12:45:53.509100 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:53.509107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:53.509134 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:45:53.509192 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:53.509198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:53.509400 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:53.509577 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:53.510747 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.511763 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.513213 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:53.513789 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:53.513907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:53.518010 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:53.518023 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:53.518038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:53.518075 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:53.520616 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.521434 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.521442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.521444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.521571 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:53.521928 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:53.521931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:53.521933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:53.522268 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:53.522552 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:53.522883 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 12:45:53.523133 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":347,"threads_started":1}
I20260924 12:45:53.524772 2877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:53.608115 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 2754
I20260924 12:45:53.620781 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 2393
I20260924 12:45:53.622006 2455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:53.721745 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 2393
2026-09-24T12:45:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3999 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2904](info): set up 2 sockets
Sep 24 12:45:53 dist-test-slave-9tm1 krb5kdc[2904](info): commencing operation
krb5kdc: starting...
W20260924 12:45:54.741117 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:45:54 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:45:54.913128 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44957
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:44957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:55.233500 2917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:55.233824 2917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:55.233955 2917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:55.241727 2917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:45:55.241865 2917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:55.241915 2917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:55.241990 2917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:45:55.242046 2917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:45:55.252322 2917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:44957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:55.253751 2917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:55.255137 2917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:55.267802 2923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:55.267813 2922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:55.268062 2925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:55.268621 2917 server_base.cc:1034] running on GCE node
I20260924 12:45:55.269476 2917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:55.270967 2917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:55.272208 2917 hybrid_clock.cc:648] HybridClock initialized: now 1790253955272128 us; error 64 us; skew 500 ppm
I20260924 12:45:55.272593 2917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:55 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253955, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:55.279403 2917 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:45:55.281177 2917 webserver.cc:460] Webserver started at http://127.2.81.126:34799/ using document root <none> and password file <none>
I20260924 12:45:55.281906 2917 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:55.282024 2917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:55.282326 2917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:55.285426 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "139cc0b462fd4a7c979ddf1faad082c1"
format_stamp: "Formatted at 2026-09-24 12:45:55 on dist-test-slave-9tm1"
server_key: "c80b0975e0cef4d3a271c1795c0697e7"
server_key_iv: "df34527646e53651a7eeaea06b9dd9ba"
server_key_version: "encryptionkey@0"
I20260924 12:45:55.286185 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "139cc0b462fd4a7c979ddf1faad082c1"
format_stamp: "Formatted at 2026-09-24 12:45:55 on dist-test-slave-9tm1"
server_key: "c80b0975e0cef4d3a271c1795c0697e7"
server_key_iv: "df34527646e53651a7eeaea06b9dd9ba"
server_key_version: "encryptionkey@0"
I20260924 12:45:55.291541 2917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 12:45:55.295179 2932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:55.296566 2917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 12:45:55.296733 2917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "139cc0b462fd4a7c979ddf1faad082c1"
format_stamp: "Formatted at 2026-09-24 12:45:55 on dist-test-slave-9tm1"
server_key: "c80b0975e0cef4d3a271c1795c0697e7"
server_key_iv: "df34527646e53651a7eeaea06b9dd9ba"
server_key_version: "encryptionkey@0"
I20260924 12:45:55.296932 2917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:55.309506 2917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:55.313817 2917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:55.314095 2917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:55.332159 2917 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:44957
I20260924 12:45:55.332190 2983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:44957 every 8 connection(s)
I20260924 12:45:55.333623 2917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:45:55.337435 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:55.339382 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 2917
I20260924 12:45:55.339594 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:45:55.340261 2373 external_mini_cluster.cc:1428] Setting key e221235fcae4def9885beb53762cbdcd
I20260924 12:45:55.349653 2984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1: Bootstrap starting.
I20260924 12:45:55.353888 2984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:55.355204 2984 log.cc:824] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:45:55 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:55.358208 2984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1: No bootstrap required, opened a new log
I20260924 12:45:55.364208 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:55.342371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.2.81.126:44957)
0924 12:45:55.343201 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:55.343214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:55.343259 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:45:55.344949 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:55.344995 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:55.345436 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:55.345806 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:55.347544 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:55.348953 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:55.350791 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:55.351730 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:55.351889 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:55.356619 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:55.356628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:55.356646 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:55.356690 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:55.359830 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:55.360644 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:55.360652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:55.360659 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:55.360854 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:55.361265 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:55.361268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:55.361270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:55.361838 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:55.362112 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:55.362642 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 12:45:55.363092 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":373,"threads_started":1}
I20260924 12:45:55.366192 2984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cc0b462fd4a7c979ddf1faad082c1" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44957 } }
I20260924 12:45:55.366609 2984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:55.366707 2984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139cc0b462fd4a7c979ddf1faad082c1, State: Initialized, Role: FOLLOWER
I20260924 12:45:55.367297 2984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cc0b462fd4a7c979ddf1faad082c1" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44957 } }
I20260924 12:45:55.367530 2984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:45:55.367648 2984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:45:55.367799 2984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:55.369997 2984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cc0b462fd4a7c979ddf1faad082c1" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44957 } }
I20260924 12:45:55.370524 2984 leader_election.cc:302] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 139cc0b462fd4a7c979ddf1faad082c1; no voters:
I20260924 12:45:55.371096 2984 leader_election.cc:288] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:45:55.371253 2989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:45:55.371930 2989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [term 1 LEADER]: Becoming Leader. State: Replica: 139cc0b462fd4a7c979ddf1faad082c1, State: Running, Role: LEADER
I20260924 12:45:55.372465 2989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cc0b462fd4a7c979ddf1faad082c1" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44957 } }
I20260924 12:45:55.373116 2984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:45:55.375469 2991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139cc0b462fd4a7c979ddf1faad082c1. Latest consensus state: current_term: 1 leader_uuid: "139cc0b462fd4a7c979ddf1faad082c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cc0b462fd4a7c979ddf1faad082c1" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44957 } } }
I20260924 12:45:55.377348 2991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:55.375366 2990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "139cc0b462fd4a7c979ddf1faad082c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cc0b462fd4a7c979ddf1faad082c1" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44957 } } }
I20260924 12:45:55.377503 2990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1 [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:55.382505 2999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:45:55.389120 2999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:45:55.398823 2999 catalog_manager.cc:1347] Generated new cluster ID: d38eb7486c0d46e9ae28bdbba5bc710d
I20260924 12:45:55.398953 2999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:45:55.408902 2999 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:45:55.410434 2999 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:45:55.417235 2999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 139cc0b462fd4a7c979ddf1faad082c1: Generated new TSK 0
I20260924 12:45:55.418409 2999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:45:55.483480 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44957
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:55.844532 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:55.844832 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:55.844954 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:55.853560 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:55.853736 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:55.853919 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:45:55.865823 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:55.867156 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:55.868533 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:55.882990 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:55.883049 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:55.883013 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:55.884631 3010 server_base.cc:1034] running on GCE node
I20260924 12:45:55.885421 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:55.886469 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:55.887709 3010 hybrid_clock.cc:648] HybridClock initialized: now 1790253955887655 us; error 51 us; skew 500 ppm
I20260924 12:45:55.888082 3010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:55 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253955, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:55.891575 3010 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:45:55.893268 3010 webserver.cc:460] Webserver started at http://127.2.81.65:37117/ using document root <none> and password file <none>
I20260924 12:45:55.893970 3010 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:55.894088 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:55.894379 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:55.897588 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "912ace8ac440486cb0e1b00aba52ba0a"
format_stamp: "Formatted at 2026-09-24 12:45:55 on dist-test-slave-9tm1"
server_key: "787c55ab619cfa297071ef4481a41048"
server_key_iv: "ee7b9afb8003ea432e0d074ce4cd61aa"
server_key_version: "encryptionkey@0"
I20260924 12:45:55.898437 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "912ace8ac440486cb0e1b00aba52ba0a"
format_stamp: "Formatted at 2026-09-24 12:45:55 on dist-test-slave-9tm1"
server_key: "787c55ab619cfa297071ef4481a41048"
server_key_iv: "ee7b9afb8003ea432e0d074ce4cd61aa"
server_key_version: "encryptionkey@0"
I20260924 12:45:55.904178 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:45:55.908001 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:55.909582 3010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:45:55.909754 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "912ace8ac440486cb0e1b00aba52ba0a"
format_stamp: "Formatted at 2026-09-24 12:45:55 on dist-test-slave-9tm1"
server_key: "787c55ab619cfa297071ef4481a41048"
server_key_iv: "ee7b9afb8003ea432e0d074ce4cd61aa"
server_key_version: "encryptionkey@0"
I20260924 12:45:55.909976 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:55.925246 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:55.929952 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:55.930231 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:55.931219 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:55.932554 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:55.932628 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:55.932749 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:55.932803 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:55.964242 3010 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:32829
I20260924 12:45:55.964326 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:32829 every 8 connection(s)
I20260924 12:45:55.965610 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:45:55.972067 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3010
I20260924 12:45:55.972282 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:45:55.972875 2373 external_mini_cluster.cc:1428] Setting key 52567f814bb6d0035a5bc56eab8e3a62
Sep 24 12:45:55 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253955, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:55.986194 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:55.968107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:33475 (local address 127.2.81.126:44957)
0924 12:45:55.968516 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:55.968523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:55.969587 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 12:45:55.971489 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:55.971494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:55.971579 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:55.971700 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:55.973505 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:55.974480 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:55.976112 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:55.976507 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:55.976616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:55.980385 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:55.980391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:55.980394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:55.980431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:55.983001 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:55.983816 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:55.983820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:55.983822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:55.983947 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:55.984390 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:55.984394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:55.984396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:55.984660 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:55.984793 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:55.985610 (+ 817us) server_negotiation.cc:299] Negotiation successful
0924 12:45:55.985841 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":186,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:45:55.987277 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:55.968660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44957 (local address 127.2.81.65:33475)
0924 12:45:55.969422 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:55.969465 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:55.971138 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:55.971987 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:55.971997 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:55.972601 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:45:55.973287 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:55.973307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:55.974683 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:55.974687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:55.975982 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:55.975994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:55.976756 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:55.976764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:55.977042 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:55.977683 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:45:55.977706 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:45:55.980087 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:45:55.983193 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:55.983203 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:55.983218 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:55.983673 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:55.984058 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:55.984064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:55.984067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:55.984271 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:55.984791 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:45:55.984800 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:45:55.985211 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 12:45:55.985472 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 12:45:55.986139 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":319,"threads_started":1}
I20260924 12:45:55.989073 3138 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44957
I20260924 12:45:55.989439 3138 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:45:55.990268 3138 heartbeater.cc:511] Master 127.2.81.126:44957 requested a full tablet report, sending...
I20260924 12:45:55.992899 2949 ts_manager.cc:194] Registered new tserver with Master: 912ace8ac440486cb0e1b00aba52ba0a (127.2.81.65:32829)
I20260924 12:45:55.994997 2949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:33475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:45:56.015429 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44957
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:56.336894 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:56.337356 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:56.337512 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:56.349696 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:56.349826 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:56.349983 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:45:56.360687 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:56.362305 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:56.363968 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:56.377408 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:56.377398 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:56.377674 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:56.378206 3144 server_base.cc:1034] running on GCE node
I20260924 12:45:56.378820 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:56.379806 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:56.381060 3144 hybrid_clock.cc:648] HybridClock initialized: now 1790253956380996 us; error 55 us; skew 500 ppm
I20260924 12:45:56.381405 3144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:56 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:56.384569 3144 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:45:56.386199 3144 webserver.cc:460] Webserver started at http://127.2.81.66:36171/ using document root <none> and password file <none>
I20260924 12:45:56.386929 3144 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:56.387037 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:56.387338 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:56.390615 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "fe3151e7d49b4fcd95e3cec24cab4f85"
format_stamp: "Formatted at 2026-09-24 12:45:56 on dist-test-slave-9tm1"
server_key: "b5c5769d10afd214bf9de6c533574618"
server_key_iv: "891ee8180eaa9ab50567a13d26992a8b"
server_key_version: "encryptionkey@0"
I20260924 12:45:56.391551 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "fe3151e7d49b4fcd95e3cec24cab4f85"
format_stamp: "Formatted at 2026-09-24 12:45:56 on dist-test-slave-9tm1"
server_key: "b5c5769d10afd214bf9de6c533574618"
server_key_iv: "891ee8180eaa9ab50567a13d26992a8b"
server_key_version: "encryptionkey@0"
I20260924 12:45:56.397223 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:45:56.401170 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:56.402828 3144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:45:56.403012 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "fe3151e7d49b4fcd95e3cec24cab4f85"
format_stamp: "Formatted at 2026-09-24 12:45:56 on dist-test-slave-9tm1"
server_key: "b5c5769d10afd214bf9de6c533574618"
server_key_iv: "891ee8180eaa9ab50567a13d26992a8b"
server_key_version: "encryptionkey@0"
I20260924 12:45:56.403223 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:56.423789 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:56.428164 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:56.428434 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:56.429454 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:56.430806 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:56.430881 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:56.430977 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:56.431025 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:56.462033 3144 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:38721
I20260924 12:45:56.462076 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:38721 every 8 connection(s)
I20260924 12:45:56.463263 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:45:56.472695 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3144
I20260924 12:45:56.472898 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:45:56.473325 2373 external_mini_cluster.cc:1428] Setting key 9fef5cb73a85f83e95b7ccef197d6c32
Sep 24 12:45:56 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:56.485574 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:56.465761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:52625 (local address 127.2.81.126:44957)
0924 12:45:56.465939 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:56.465944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:56.467077 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 12:45:56.468907 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:56.468912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:56.469026 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:56.469108 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:56.470692 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.471419 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.473514 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.473998 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.474115 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:56.478434 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:56.478440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:56.478443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:56.478480 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:56.481543 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:56.482563 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:56.482567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:56.482570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:56.482786 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:56.483403 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:56.483407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:56.483409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:56.483734 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:56.483855 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:56.484823 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 12:45:56.485083 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:45:56.486732 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:56.466237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44957 (local address 127.2.81.66:52625)
0924 12:45:56.466920 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:56.466963 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:56.468628 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:56.469374 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:56.469383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:56.469907 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:45:56.470507 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:56.470525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.471590 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.471594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:56.473341 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:56.473359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.474263 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.474271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:56.474501 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:56.475225 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:45:56.475254 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:45:56.478153 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:45:56.481728 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:56.481739 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:56.481759 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:56.482400 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:56.482959 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:56.482964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:56.482969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:56.483244 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:56.483868 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:45:56.483877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:45:56.484352 (+ 475us) client_negotiation.cc:769] Sending connection context
0924 12:45:56.484688 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 12:45:56.485291 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260924 12:45:56.488423 3272 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44957
I20260924 12:45:56.488770 3272 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:56.489753 3272 heartbeater.cc:511] Master 127.2.81.126:44957 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:45:56.491426 2949 ts_manager.cc:194] Registered new tserver with Master: fe3151e7d49b4fcd95e3cec24cab4f85 (127.2.81.66:38721)
I20260924 12:45:56.492229 2949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:52625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:45:56.516624 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44957
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:56.839550 3277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:56.839833 3277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:56.839944 3277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:56.847254 3277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:56.847384 3277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:56.847530 3277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:45:56.857616 3277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:56.858897 3277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:56.860213 3277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:56.872673 3282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:56.872705 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:56.872733 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:56.873414 3277 server_base.cc:1034] running on GCE node
I20260924 12:45:56.874111 3277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:56.875083 3277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:56.876312 3277 hybrid_clock.cc:648] HybridClock initialized: now 1790253956876208 us; error 72 us; skew 500 ppm
I20260924 12:45:56.876629 3277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:56 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:56.879781 3277 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:45:56.881317 3277 webserver.cc:460] Webserver started at http://127.2.81.67:41989/ using document root <none> and password file <none>
I20260924 12:45:56.881999 3277 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:56.882112 3277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:56.882400 3277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:56.885298 3277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "0363e9e11f124211a9cd4e895dfeeefc"
format_stamp: "Formatted at 2026-09-24 12:45:56 on dist-test-slave-9tm1"
server_key: "7f767bb36c626bd2fc7ea5c7afdbddff"
server_key_iv: "7d91fb24b5c34790f83ab3d2044e646a"
server_key_version: "encryptionkey@0"
I20260924 12:45:56.886035 3277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "0363e9e11f124211a9cd4e895dfeeefc"
format_stamp: "Formatted at 2026-09-24 12:45:56 on dist-test-slave-9tm1"
server_key: "7f767bb36c626bd2fc7ea5c7afdbddff"
server_key_iv: "7d91fb24b5c34790f83ab3d2044e646a"
server_key_version: "encryptionkey@0"
I20260924 12:45:56.891077 3277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:45:56.894901 3292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:56.896545 3277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:45:56.896729 3277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "0363e9e11f124211a9cd4e895dfeeefc"
format_stamp: "Formatted at 2026-09-24 12:45:56 on dist-test-slave-9tm1"
server_key: "7f767bb36c626bd2fc7ea5c7afdbddff"
server_key_iv: "7d91fb24b5c34790f83ab3d2044e646a"
server_key_version: "encryptionkey@0"
I20260924 12:45:56.896948 3277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:56.916247 3277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:56.920799 3277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:56.921140 3277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:56.922155 3277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:45:56.923583 3277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:45:56.923662 3277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:56.923777 3277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:45:56.923826 3277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:56.954422 3277 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:46629
I20260924 12:45:56.954469 3404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:46629 every 8 connection(s)
I20260924 12:45:56.955699 3277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:45:56.962541 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3277
I20260924 12:45:56.962724 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:45:56.963137 2373 external_mini_cluster.cc:1428] Setting key 555c5199464841f8d6548fed85f1f7d5
Sep 24 12:45:56 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:56.977056 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:56.958427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:41211 (local address 127.2.81.126:44957)
0924 12:45:56.958635 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:56.958639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:56.959869 (+ 1230us) server_negotiation.cc:407] Connection header received
0924 12:45:56.961845 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:56.961851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:56.961969 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:56.962058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:56.963921 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.964953 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.967159 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.967768 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.967899 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:56.971338 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:56.971345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:56.971349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:56.971385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:56.973994 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:56.974751 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:56.974757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:56.974761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:56.974896 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:56.975299 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:56.975304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:56.975307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:56.975594 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:56.975734 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:56.976450 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 12:45:56.976670 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:45:56.978199 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:56.958937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44957 (local address 127.2.81.67:41211)
0924 12:45:56.959694 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:56.959744 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:56.961535 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:56.962338 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:56.962348 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:56.962955 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:45:56.963635 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:56.963658 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.965122 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.965128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:56.966750 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:56.966763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:56.967992 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:56.967998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:56.968197 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:56.968865 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:45:56.968891 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:45:56.971081 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:45:56.974136 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:56.974146 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:56.974164 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:56.974637 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:56.975000 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:45:56.975005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:45:56.975009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:45:56.975194 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:45:56.975713 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:45:56.975721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:45:56.976072 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 12:45:56.976373 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 12:45:56.976922 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":291,"threads_started":1}
I20260924 12:45:56.979640 3405 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44957
I20260924 12:45:56.979929 3405 heartbeater.cc:464] Registering TS with master...
I20260924 12:45:56.980611 3405 heartbeater.cc:511] Master 127.2.81.126:44957 requested a full tablet report, sending...
I20260924 12:45:56.982049 2948 ts_manager.cc:194] Registered new tserver with Master: 0363e9e11f124211a9cd4e895dfeeefc (127.2.81.67:46629)
I20260924 12:45:56.982939 2948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:41211
I20260924 12:45:56.992285 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:45:56.998415 3138 heartbeater.cc:503] Master 127.2.81.126:44957 was elected leader, sending a full tablet report...
I20260924 12:45:57.012326 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:56.998734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.2.81.126:44957)
0924 12:45:56.998924 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:56.998930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:56.999331 (+ 401us) server_negotiation.cc:407] Connection header received
0924 12:45:56.999496 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:56.999501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:56.999571 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:56.999669 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.001478 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.002308 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.003769 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.004201 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.004311 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.006698 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.006704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.006707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.006743 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.009645 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.010457 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.010461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.010463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.010598 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.010919 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.010922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.010924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.011249 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.011473 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.011847 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.012021 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:45:57.019109 2948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:45:57.026371 2948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:45:57.057447 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.039115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:38721 (local address 127.0.0.1:60836)
0924 12:45:57.045563 (+ 6448us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:57.045609 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:57.045813 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:57.049183 (+ 3370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:57.049187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:57.049209 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:45:57.049576 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.049588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.051432 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.051437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.054427 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.054444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.056166 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.056172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.056640 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.056697 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 12:45:57.056831 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 12:45:57.057058 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6160,"thread_start_us":5973,"threads_started":1}
I20260924 12:45:57.058239 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.039299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.2.81.66:38721)
0924 12:45:57.048013 (+ 8714us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.048022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.048050 (+ 28us) server_negotiation.cc:407] Connection header received
0924 12:45:57.048142 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.048149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.048400 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.048600 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:45:57.049764 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.051268 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.054645 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.055962 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.056105 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.056307 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.057739 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.057930 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8556,"thread_start_us":8454,"threads_started":1}
I20260924 12:45:57.058851 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.039641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:32829 (local address 127.0.0.1:50900)
0924 12:45:57.046319 (+ 6678us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:57.046339 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:57.046465 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:57.048214 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:57.048222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:57.048252 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:45:57.049200 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.049212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.052687 (+ 3475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.052693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.055239 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.055252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.057411 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.057418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.057592 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.057623 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:45:57.057695 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 12:45:57.057834 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6528,"thread_start_us":6379,"threads_started":1}
I20260924 12:45:57.058995 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.040350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.2.81.65:32829)
0924 12:45:57.047026 (+ 6676us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.047035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.047065 (+ 30us) server_negotiation.cc:407] Connection header received
0924 12:45:57.047166 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.047175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.047467 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.049423 (+ 1956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:45:57.049662 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.052512 (+ 2850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.055568 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.057164 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.058235 (+ 1071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.058460 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.058546 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.058686 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6519,"thread_start_us":6374,"threads_started":1}
I20260924 12:45:57.062383 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.048172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:46629 (local address 127.0.0.1:42428)
0924 12:45:57.054664 (+ 6492us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:57.054684 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:57.054826 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:57.055436 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:57.055440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:57.055464 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:45:57.055803 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.055812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.057625 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.057630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.060296 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.060310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.061648 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.061655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.061816 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.061847 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:45:57.061935 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 12:45:57.062063 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6359,"thread_start_us":6204,"threads_started":1}
I20260924 12:45:57.062425 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.046808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.2.81.67:46629)
0924 12:45:57.054130 (+ 7322us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.054139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.054799 (+ 660us) server_negotiation.cc:407] Connection header received
0924 12:45:57.054979 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.054987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.055263 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.055461 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:45:57.055949 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.057431 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.060486 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.061458 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.061583 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.061839 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.061982 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.062129 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7139,"thread_start_us":6967,"threads_started":1}
I20260924 12:45:57.062500 3207 tablet_service.cc:1467] Processing CreateTablet for tablet b52e8fcf02ab4ad59acfad4f563a1c01 (DEFAULT_TABLE table=test-table [id=b980fe5bbcac4dffb827997512def413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:45:57.062500 3073 tablet_service.cc:1467] Processing CreateTablet for tablet b52e8fcf02ab4ad59acfad4f563a1c01 (DEFAULT_TABLE table=test-table [id=b980fe5bbcac4dffb827997512def413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:45:57.063972 3207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52e8fcf02ab4ad59acfad4f563a1c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:57.063972 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52e8fcf02ab4ad59acfad4f563a1c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:57.066275 3340 tablet_service.cc:1467] Processing CreateTablet for tablet b52e8fcf02ab4ad59acfad4f563a1c01 (DEFAULT_TABLE table=test-table [id=b980fe5bbcac4dffb827997512def413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:45:57.067649 3340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52e8fcf02ab4ad59acfad4f563a1c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:45:57.079154 3423 tablet_bootstrap.cc:492] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: Bootstrap starting.
I20260924 12:45:57.081409 3424 tablet_bootstrap.cc:492] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: Bootstrap starting.
I20260924 12:45:57.084692 3424 tablet_bootstrap.cc:654] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:57.085027 3423 tablet_bootstrap.cc:654] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:57.085851 3424 log.cc:824] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:57.086395 3423 log.cc:824] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:57.088290 3425 tablet_bootstrap.cc:492] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: Bootstrap starting.
I20260924 12:45:57.088928 3424 tablet_bootstrap.cc:492] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: No bootstrap required, opened a new log
I20260924 12:45:57.089206 3424 ts_tablet_manager.cc:1397] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260924 12:45:57.090219 3423 tablet_bootstrap.cc:492] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: No bootstrap required, opened a new log
I20260924 12:45:57.090528 3423 ts_tablet_manager.cc:1397] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260924 12:45:57.092023 3425 tablet_bootstrap.cc:654] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:57.093327 3425 log.cc:824] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:57.097635 3425 tablet_bootstrap.cc:492] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: No bootstrap required, opened a new log
I20260924 12:45:57.097952 3425 ts_tablet_manager.cc:1397] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 12:45:57.097697 3424 raft_consensus.cc:348] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.098091 3424 raft_consensus.cc:374] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:57.098199 3424 raft_consensus.cc:729] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912ace8ac440486cb0e1b00aba52ba0a, State: Initialized, Role: FOLLOWER
I20260924 12:45:57.098794 3424 consensus_queue.cc:260] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.101025 3424 ts_tablet_manager.cc:1428] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260924 12:45:57.100805 3423 raft_consensus.cc:348] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.101434 3423 raft_consensus.cc:374] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:57.101540 3423 raft_consensus.cc:729] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3151e7d49b4fcd95e3cec24cab4f85, State: Initialized, Role: FOLLOWER
I20260924 12:45:57.102195 3423 consensus_queue.cc:260] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.103880 3272 heartbeater.cc:503] Master 127.2.81.126:44957 was elected leader, sending a full tablet report...
I20260924 12:45:57.104672 3423 ts_tablet_manager.cc:1428] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 12:45:57.108430 3425 raft_consensus.cc:348] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.108835 3425 raft_consensus.cc:374] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:57.108949 3425 raft_consensus.cc:729] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0363e9e11f124211a9cd4e895dfeeefc, State: Initialized, Role: FOLLOWER
I20260924 12:45:57.109611 3425 consensus_queue.cc:260] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.110695 3405 heartbeater.cc:503] Master 127.2.81.126:44957 was elected leader, sending a full tablet report...
I20260924 12:45:57.111469 3425 ts_tablet_manager.cc:1428] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 12:45:57.184770 3430 raft_consensus.cc:482] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:45:57.185194 3430 raft_consensus.cc:504] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.186885 3430 leader_election.cc:288] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912ace8ac440486cb0e1b00aba52ba0a (127.2.81.65:32829), 0363e9e11f124211a9cd4e895dfeeefc (127.2.81.67:46629)
I20260924 12:45:57.193593 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.187296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:53841 (local address 127.2.81.65:32829)
0924 12:45:57.187496 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.187500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.188067 (+ 567us) server_negotiation.cc:407] Connection header received
0924 12:45:57.188275 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.188280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.188357 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.188467 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:45:57.189412 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.190138 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.191889 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.192670 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.192791 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.192900 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.193229 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.193352 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 12:45:57.193684 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.187270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:32829 (local address 127.2.81.66:53841)
0924 12:45:57.187961 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:57.187979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:57.188117 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:57.188591 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:57.188596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:57.188624 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:45:57.189261 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.189272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.190266 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.190271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.191724 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.191736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.192886 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.192892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.193080 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.193106 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 12:45:57.193192 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 12:45:57.193333 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":374,"threads_started":1}
I20260924 12:45:57.194046 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.188264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:42639 (local address 127.2.81.67:46629)
0924 12:45:57.188495 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.188501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.189233 (+ 732us) server_negotiation.cc:407] Connection header received
0924 12:45:57.189398 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.189402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.189457 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.189526 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:45:57.189995 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.190695 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.192352 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.193157 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.193268 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.193362 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.193682 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.193804 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 12:45:57.194527 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.188114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:46629 (local address 127.2.81.66:42639)
0924 12:45:57.189129 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0924 12:45:57.189143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 12:45:57.189268 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:45:57.189550 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:45:57.189554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:45:57.189575 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:45:57.189840 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.189848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.190823 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.190827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.192206 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:45:57.192216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.193348 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.193355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:45:57.193547 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.193570 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 12:45:57.193652 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 12:45:57.193864 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":1,"thread_start_us":256,"threads_started":1}
I20260924 12:45:57.194803 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e8fcf02ab4ad59acfad4f563a1c01" candidate_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912ace8ac440486cb0e1b00aba52ba0a" is_pre_election: true
I20260924 12:45:57.195497 3093 raft_consensus.cc:2454] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3151e7d49b4fcd95e3cec24cab4f85 in term 0.
I20260924 12:45:57.195538 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e8fcf02ab4ad59acfad4f563a1c01" candidate_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0363e9e11f124211a9cd4e895dfeeefc" is_pre_election: true
I20260924 12:45:57.196156 3360 raft_consensus.cc:2454] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3151e7d49b4fcd95e3cec24cab4f85 in term 0.
I20260924 12:45:57.197037 3161 leader_election.cc:302] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912ace8ac440486cb0e1b00aba52ba0a, fe3151e7d49b4fcd95e3cec24cab4f85; no voters:
I20260924 12:45:57.197417 3430 raft_consensus.cc:2781] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:45:57.197570 3430 raft_consensus.cc:482] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:45:57.197685 3430 raft_consensus.cc:3037] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:57.199921 3430 raft_consensus.cc:504] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
I20260924 12:45:57.200552 3430 leader_election.cc:288] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [CANDIDATE]: Term 1 election: Requested vote from peers 912ace8ac440486cb0e1b00aba52ba0a (127.2.81.65:32829), 0363e9e11f124211a9cd4e895dfeeefc (127.2.81.67:46629)
I20260924 12:45:57.201138 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e8fcf02ab4ad59acfad4f563a1c01" candidate_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912ace8ac440486cb0e1b00aba52ba0a"
I20260924 12:45:57.201331 3093 raft_consensus.cc:3037] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:57.201709 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52e8fcf02ab4ad59acfad4f563a1c01" candidate_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0363e9e11f124211a9cd4e895dfeeefc"
I20260924 12:45:57.201918 3360 raft_consensus.cc:3037] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:57.204537 3093 raft_consensus.cc:2454] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3151e7d49b4fcd95e3cec24cab4f85 in term 1.
I20260924 12:45:57.204895 3360 raft_consensus.cc:2454] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3151e7d49b4fcd95e3cec24cab4f85 in term 1.
I20260924 12:45:57.205104 3161 leader_election.cc:302] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912ace8ac440486cb0e1b00aba52ba0a, fe3151e7d49b4fcd95e3cec24cab4f85; no voters:
I20260924 12:45:57.205415 3430 raft_consensus.cc:2781] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:45:57.205864 3430 raft_consensus.cc:686] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [term 1 LEADER]: Becoming Leader. State: Replica: fe3151e7d49b4fcd95e3cec24cab4f85, State: Running, Role: LEADER
I20260924 12:45:57.206343 3430 consensus_queue.cc:237] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } }
W20260924 12:45:57.207296 3406 tablet.cc:2369] T b52e8fcf02ab4ad59acfad4f563a1c01 P 0363e9e11f124211a9cd4e895dfeeefc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:45:57.211325 2949 catalog_manager.cc:5495] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3151e7d49b4fcd95e3cec24cab4f85 (127.2.81.66). New cstate: current_term: 1 leader_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3151e7d49b4fcd95e3cec24cab4f85" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912ace8ac440486cb0e1b00aba52ba0a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0363e9e11f124211a9cd4e895dfeeefc" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 46629 } health_report { overall_health: UNKNOWN } } }
W20260924 12:45:57.215170 3273 tablet.cc:2369] T b52e8fcf02ab4ad59acfad4f563a1c01 P fe3151e7d49b4fcd95e3cec24cab4f85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:45:57.217862 3139 tablet.cc:2369] T b52e8fcf02ab4ad59acfad4f563a1c01 P 912ace8ac440486cb0e1b00aba52ba0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:45:57.251984 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.235486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.2.81.65:32829)
0924 12:45:57.235647 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.235651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.236178 (+ 527us) server_negotiation.cc:407] Connection header received
0924 12:45:57.236341 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.236347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.236418 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.236503 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.238448 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.239233 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.240603 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.241116 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.241220 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.245256 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.245271 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.245288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.245334 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.248599 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.249824 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.249833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.249835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.250003 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.250482 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.250487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.250489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.250898 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.251137 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.251494 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.251676 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:45:57.296056 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.282290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.2.81.65:32829)
0924 12:45:57.282513 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.282518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.282538 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:45:57.282594 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.282598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.282656 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.282741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.284070 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.284812 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.286147 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.286685 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.286795 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.290537 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.290542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.290546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.290594 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.293213 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.293903 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.293908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.293910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.294049 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.294666 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.294672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.294675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.294993 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.295190 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.295542 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.295723 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:45:57.340372 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.324996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.2.81.65:32829)
0924 12:45:57.325225 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.325231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.325574 (+ 343us) server_negotiation.cc:407] Connection header received
0924 12:45:57.325797 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.325801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.325865 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.325950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.327278 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.327974 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.329250 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.329685 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.329809 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.334916 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.334922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.334925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.334970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.337586 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.338450 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.338454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.338457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.338580 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.338937 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.338940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.338942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.339229 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.339428 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.339811 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.340008 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 12:45:57.342197 3053 tablet_service.cc:2908] Rejecting scan request for tablet b52e8fcf02ab4ad59acfad4f563a1c01: Uninitialized: clean time has not yet been initialized
I20260924 12:45:57.345309 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3010
I20260924 12:45:57.361407 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.346633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.2.81.65:32829)
0924 12:45:57.346821 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.346827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.346850 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:45:57.346925 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.346931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.347006 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.347105 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.348432 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.349441 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.350732 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.351178 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.351316 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.354502 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.354509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.354513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.354556 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.357864 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.358866 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.358871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.358874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.359016 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.359554 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.359559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.359561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.360043 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.360284 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.360876 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.361158 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:45:57.362111 3133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:57.450557 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3010
I20260924 12:45:57.464308 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3144
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:45:57.482107 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.465214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.2.81.66:38721)
0924 12:45:57.465410 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.465416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.465438 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:45:57.465575 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.465581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.465663 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.465771 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.466987 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.468126 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.469688 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.470346 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.470470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.475496 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.475508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.475524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.475578 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.478822 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.479784 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.479792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.479794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.479968 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.480523 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.480527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.480529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.480946 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.481240 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.481599 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.481820 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:45:57.482890 3267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:57.572870 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3144
I20260924 12:45:57.586460 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3277
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[2904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:45:57.602545 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:57.587262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.2.81.67:46629)
0924 12:45:57.588080 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:57.588087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:57.588108 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:45:57.588162 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:57.588165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:57.588309 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:57.588456 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:57.589758 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.590518 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.591861 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:57.592294 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:57.592430 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:57.596554 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:57.596571 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:57.596587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:57.596630 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:57.599618 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.600461 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.600469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.600471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.600602 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:57.600964 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:57.600968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:57.600999 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:57.601390 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:57.601657 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:57.602017 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 12:45:57.602212 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632}
I20260924 12:45:57.603263 3400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:57.692287 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3277
I20260924 12:45:57.705242 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 2917
I20260924 12:45:57.706441 2979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:45:57.816996 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 2917
2026-09-24T12:45:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4095 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[3453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[3453](info): set up 2 sockets
Sep 24 12:45:57 dist-test-slave-9tm1 krb5kdc[3453](info): commencing operation
krb5kdc: starting...
W20260924 12:45:58.840952 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:45:58 dist-test-slave-9tm1 krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:45:59.029698 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41541
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:41541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:59.354280 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:59.354605 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:59.354720 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:59.362215 3466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:45:59.362339 3466 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 12:45:59.362377 3466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:59.362442 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:59.362516 3466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:45:59.362601 3466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:45:59.373431 3466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:41541
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:59.374771 3466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:59.376106 3466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:59.388799 3471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:59.388793 3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:59.388723 3474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:59.389493 3466 server_base.cc:1034] running on GCE node
I20260924 12:45:59.390343 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:59.391975 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:59.393250 3466 hybrid_clock.cc:648] HybridClock initialized: now 1790253959393175 us; error 47 us; skew 500 ppm
I20260924 12:45:59.393631 3466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:59 dist-test-slave-9tm1 krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253959, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:59.396903 3466 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:45:59.398531 3466 webserver.cc:460] Webserver started at http://127.2.81.126:40415/ using document root <none> and password file <none>
I20260924 12:45:59.399247 3466 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:59.399346 3466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:59.399672 3466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:59.402697 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "4b9c31785f144f1bbbceb22588812e1c"
format_stamp: "Formatted at 2026-09-24 12:45:59 on dist-test-slave-9tm1"
server_key: "022ac4c749fde72062a922b938dc5ae2"
server_key_iv: "994118a5728315abe126476c6152718f"
server_key_version: "encryptionkey@0"
I20260924 12:45:59.403522 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "4b9c31785f144f1bbbceb22588812e1c"
format_stamp: "Formatted at 2026-09-24 12:45:59 on dist-test-slave-9tm1"
server_key: "022ac4c749fde72062a922b938dc5ae2"
server_key_iv: "994118a5728315abe126476c6152718f"
server_key_version: "encryptionkey@0"
I20260924 12:45:59.409198 3466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 12:45:59.413004 3481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:45:59.414696 3466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:45:59.414896 3466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "4b9c31785f144f1bbbceb22588812e1c"
format_stamp: "Formatted at 2026-09-24 12:45:59 on dist-test-slave-9tm1"
server_key: "022ac4c749fde72062a922b938dc5ae2"
server_key_iv: "994118a5728315abe126476c6152718f"
server_key_version: "encryptionkey@0"
I20260924 12:45:59.415112 3466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:45:59.447883 3466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:45:59.452678 3466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:45:59.452958 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:45:59.474432 3466 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:41541
I20260924 12:45:59.474449 3542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:41541 every 8 connection(s)
I20260924 12:45:59.475881 3466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:45:59.477313 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3466
I20260924 12:45:59.477542 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:45:59.478312 2373 external_mini_cluster.cc:1428] Setting key 2800eeed63d7cd0a4883089312f670c8
I20260924 12:45:59.480855 3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:45:59 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:45:59.494251 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: Bootstrap starting.
I20260924 12:45:59.498494 3543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: Neither blocks nor log segments found. Creating new log.
I20260924 12:45:59.499689 3543 log.cc:824] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:45:59.503283 3543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: No bootstrap required, opened a new log
I20260924 12:45:59.503474 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:45:59.480472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.2.81.126:41541)
0924 12:45:59.481267 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 12:45:59.481279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:45:59.481322 (+ 43us) server_negotiation.cc:407] Connection header received
0924 12:45:59.482957 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:45:59.482979 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:45:59.483407 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:45:59.483771 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:45:59.485567 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:59.487068 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:59.488768 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:45:59.489581 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:45:59.489718 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:45:59.494688 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:45:59.494699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:45:59.494721 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:45:59.494784 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:45:59.498674 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:59.499427 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:59.499436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:59.499446 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:59.499665 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:45:59.500148 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:45:59.500153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:45:59.500158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:45:59.500812 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:45:59.501106 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 12:45:59.501578 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 12:45:59.502045 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":328,"threads_started":1}
I20260924 12:45:59.511334 3543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9c31785f144f1bbbceb22588812e1c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41541 } }
I20260924 12:45:59.511778 3543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:45:59.511889 3543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9c31785f144f1bbbceb22588812e1c, State: Initialized, Role: FOLLOWER
I20260924 12:45:59.512570 3543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9c31785f144f1bbbceb22588812e1c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41541 } }
I20260924 12:45:59.512791 3543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:45:59.512917 3543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:45:59.513105 3543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:45:59.515262 3543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9c31785f144f1bbbceb22588812e1c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41541 } }
I20260924 12:45:59.515813 3543 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9c31785f144f1bbbceb22588812e1c; no voters:
I20260924 12:45:59.516458 3543 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:45:59.516561 3548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:45:59.517441 3548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [term 1 LEADER]: Becoming Leader. State: Replica: 4b9c31785f144f1bbbceb22588812e1c, State: Running, Role: LEADER
I20260924 12:45:59.517972 3548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9c31785f144f1bbbceb22588812e1c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41541 } }
I20260924 12:45:59.518651 3543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:45:59.520619 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9c31785f144f1bbbceb22588812e1c. Latest consensus state: current_term: 1 leader_uuid: "4b9c31785f144f1bbbceb22588812e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9c31785f144f1bbbceb22588812e1c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41541 } } }
I20260924 12:45:59.521126 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:59.521308 3549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9c31785f144f1bbbceb22588812e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9c31785f144f1bbbceb22588812e1c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41541 } } }
I20260924 12:45:59.521574 3549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c [sys.catalog]: This master's current role is: LEADER
I20260924 12:45:59.523792 3553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:45:59.530426 3553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:45:59.539846 3553 catalog_manager.cc:1347] Generated new cluster ID: 71348d17908b4c08be9be1fe16ef50d1
I20260924 12:45:59.539978 3553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:45:59.555930 3553 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:45:59.557372 3553 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:45:59.570228 3553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: Generated new TSK 0
I20260924 12:45:59.571215 3553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:45:59.621307 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41541
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:45:59.952652 3569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:45:59.952947 3569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:45:59.953091 3569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:45:59.960486 3569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:45:59.960603 3569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:45:59.960708 3569 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:45:59.960820 3569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:45:59.970732 3569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:45:59.972030 3569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:45:59.973430 3569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:45:59.986678 3575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:59.986730 3577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:45:59.986708 3574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:45:59.987527 3569 server_base.cc:1034] running on GCE node
I20260924 12:45:59.988183 3569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:45:59.989187 3569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:45:59.990406 3569 hybrid_clock.cc:648] HybridClock initialized: now 1790253959990330 us; error 56 us; skew 500 ppm
I20260924 12:45:59.990711 3569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:45:59 dist-test-slave-9tm1 krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253959, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:45:59.993803 3569 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:45:59.995242 3569 webserver.cc:460] Webserver started at http://127.2.81.65:35735/ using document root <none> and password file <none>
I20260924 12:45:59.995927 3569 fs_manager.cc:359] Metadata directory not provided
I20260924 12:45:59.996022 3569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:45:59.996343 3569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:45:59.999444 3569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "f021935cf998440882bc67ad94bb0bf2"
format_stamp: "Formatted at 2026-09-24 12:45:59 on dist-test-slave-9tm1"
server_key: "32ba605c8f30364bacba0526d50f42a8"
server_key_iv: "e8fae67711502af145e1e4a69df93980"
server_key_version: "encryptionkey@0"
I20260924 12:46:00.000231 3569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "f021935cf998440882bc67ad94bb0bf2"
format_stamp: "Formatted at 2026-09-24 12:45:59 on dist-test-slave-9tm1"
server_key: "32ba605c8f30364bacba0526d50f42a8"
server_key_iv: "e8fae67711502af145e1e4a69df93980"
server_key_version: "encryptionkey@0"
I20260924 12:46:00.006445 3569 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 12:46:00.010407 3584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:00.012032 3569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:00.012230 3569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "f021935cf998440882bc67ad94bb0bf2"
format_stamp: "Formatted at 2026-09-24 12:45:59 on dist-test-slave-9tm1"
server_key: "32ba605c8f30364bacba0526d50f42a8"
server_key_iv: "e8fae67711502af145e1e4a69df93980"
server_key_version: "encryptionkey@0"
I20260924 12:46:00.012555 3569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:00.029340 3569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:00.033882 3569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:00.034147 3569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:00.036849 3569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:00.036940 3569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:00.037122 3569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:00.037199 3569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:00 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253959, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:00.062060 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.037886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:56507 (local address 127.2.81.126:41541)
0924 12:46:00.038360 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:00.038365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:00.039609 (+ 1244us) server_negotiation.cc:407] Connection header received
0924 12:46:00.041950 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:00.041956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:00.042044 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:00.042111 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:00.044663 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.045680 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.048141 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.048744 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.048874 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.054659 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:00.054668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:00.054673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:00.054723 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:00.058076 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.059133 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.059140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.059144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.059269 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.059838 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.059843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.059846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.060214 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:00.060347 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:00.061367 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 12:46:00.061643 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":229,"threads_started":1}
I20260924 12:46:00.063146 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.038571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.2.81.65:56507)
0924 12:46:00.039430 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:00.039471 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:00.041553 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:00.042654 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:00.042665 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:00.043507 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:00.044449 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.044484 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.045858 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.045865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.047975 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.047991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.049043 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.049053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.049430 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.050501 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:00.050547 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:00.054287 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:00.058249 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.058263 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.058286 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.058966 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.059428 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.059434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.059438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.059706 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.060354 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:00.060362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:00.060856 (+ 494us) client_negotiation.cc:769] Sending connection context
0924 12:46:00.061242 (+ 386us) client_negotiation.cc:240] Negotiation successful
0924 12:46:00.061729 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":343,"threads_started":1}
I20260924 12:46:00.077628 3569 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:46845
I20260924 12:46:00.077723 3704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:46845 every 8 connection(s)
I20260924 12:46:00.079139 3569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:00.081290 3705 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41541
I20260924 12:46:00.081553 3705 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:00.082435 3705 heartbeater.cc:511] Master 127.2.81.126:41541 requested a full tablet report, sending...
I20260924 12:46:00.084789 3498 ts_manager.cc:194] Registered new tserver with Master: f021935cf998440882bc67ad94bb0bf2 (127.2.81.65:46845)
I20260924 12:46:00.086386 3498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:56507
I20260924 12:46:00.089037 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3569
I20260924 12:46:00.089345 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:00.089969 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.075959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:40727 (local address 127.2.81.126:41541)
0924 12:46:00.076146 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:00.076153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:00.077237 (+ 1084us) server_negotiation.cc:407] Connection header received
0924 12:46:00.077575 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:00.077580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:00.077674 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:00.077758 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:00.078808 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.079540 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.081174 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.081808 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.081946 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.084023 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:00.084028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:00.084032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:00.084065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:00.086819 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.087680 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.087686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.087689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.087849 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.088235 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.088240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.088244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.088621 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:00.088773 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:00.089403 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 12:46:00.089647 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:46:00.090103 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.075922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.2.81.65:40727)
0924 12:46:00.077099 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:00.077122 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:00.077416 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:00.077834 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:00.077839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:00.078348 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:00.078662 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.078674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.079687 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.079692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.081037 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.081053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.082009 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.082015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.082158 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.082903 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:00.082921 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:00.083807 (+ 886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:00.086992 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.086997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.087001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.087552 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.087993 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.087997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.087999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.088125 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.088816 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:00.088822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:00.089119 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 12:46:00.089301 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 12:46:00.089627 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":4,"thread_start_us":772,"threads_started":1}
I20260924 12:46:00.090139 2373 external_mini_cluster.cc:1428] Setting key 18904a76a51a1c6186902f0cff256882
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:00.130442 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41541
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:00.458709 3709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:00.459088 3709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:00.459239 3709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:00.467463 3709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:00.467614 3709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:00.467738 3709 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:00.467851 3709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:00.478060 3709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:00.479712 3709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:00.481247 3709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:00.496162 3714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:00.496213 3715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:00.496140 3717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:00.496758 3709 server_base.cc:1034] running on GCE node
I20260924 12:46:00.497622 3709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:00.498672 3709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:00.499946 3709 hybrid_clock.cc:648] HybridClock initialized: now 1790253960499881 us; error 35 us; skew 500 ppm
I20260924 12:46:00.500288 3709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:00 dist-test-slave-9tm1 krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253960, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:00.510558 3709 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:00.512444 3709 webserver.cc:460] Webserver started at http://127.2.81.66:33719/ using document root <none> and password file <none>
I20260924 12:46:00.513250 3709 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:00.513366 3709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:00.513702 3709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:00.516698 3709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "515e40103de444ee8fd9e6ac50e7586c"
format_stamp: "Formatted at 2026-09-24 12:46:00 on dist-test-slave-9tm1"
server_key: "607b9f6ade85de1f3797d6f69578413d"
server_key_iv: "7a2158890fbbb0117d8aed201bd0915e"
server_key_version: "encryptionkey@0"
I20260924 12:46:00.517642 3709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "515e40103de444ee8fd9e6ac50e7586c"
format_stamp: "Formatted at 2026-09-24 12:46:00 on dist-test-slave-9tm1"
server_key: "607b9f6ade85de1f3797d6f69578413d"
server_key_iv: "7a2158890fbbb0117d8aed201bd0915e"
server_key_version: "encryptionkey@0"
I20260924 12:46:00.523273 3709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:00.527102 3724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:00.528786 3709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:00.528957 3709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "515e40103de444ee8fd9e6ac50e7586c"
format_stamp: "Formatted at 2026-09-24 12:46:00 on dist-test-slave-9tm1"
server_key: "607b9f6ade85de1f3797d6f69578413d"
server_key_iv: "7a2158890fbbb0117d8aed201bd0915e"
server_key_version: "encryptionkey@0"
I20260924 12:46:00.529227 3709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:00.555732 3709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:00.560139 3709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:00.560412 3709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:00.563020 3709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:00.563115 3709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:00.563236 3709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:00.563292 3709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:00 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253960, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:00.595530 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.563730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:59889 (local address 127.2.81.126:41541)
0924 12:46:00.563963 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:00.563970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:00.565539 (+ 1569us) server_negotiation.cc:407] Connection header received
0924 12:46:00.568490 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:00.568497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:00.568593 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:00.568693 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:00.571592 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.572514 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.575746 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.576163 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.576277 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.584238 (+ 7961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:00.584251 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:00.584257 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:00.584418 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:00.588138 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.589448 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.589455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.589460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.589654 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.590308 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.590313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.590316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.590753 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:00.594597 (+ 3844us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:00.594829 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 12:46:00.595133 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 12:46:00.596762 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.564419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.2.81.66:59889)
0924 12:46:00.565351 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:00.565407 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:00.568060 (+ 2653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:00.569113 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:00.569130 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:00.570114 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:00.571216 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.571260 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.572745 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.572755 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.575436 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.575469 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.576455 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.576464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.576799 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.578157 (+ 1358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:00.578199 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:00.583643 (+ 5444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:00.588372 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.588387 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.588413 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.589233 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.589816 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.589823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.589827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.590151 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.590940 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:00.590951 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:00.591532 (+ 581us) client_negotiation.cc:769] Sending connection context
0924 12:46:00.591884 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 12:46:00.592568 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":326,"threads_started":1}
I20260924 12:46:00.619640 3709 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:33825
I20260924 12:46:00.619655 3843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:33825 every 8 connection(s)
I20260924 12:46:00.621080 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.606011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:47391 (local address 127.2.81.126:41541)
0924 12:46:00.606195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:00.606201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:00.606979 (+ 778us) server_negotiation.cc:407] Connection header received
0924 12:46:00.607230 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:00.607235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:00.607302 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:00.607405 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:00.608861 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.609888 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.611469 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.611994 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.612109 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.614520 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:00.614525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:00.614528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:00.614563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:00.617990 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.618858 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.618863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.618865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.619067 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:00.619517 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:00.619522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:00.619525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:00.619934 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:00.620070 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:00.620504 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 12:46:00.620719 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:46:00.621107 3709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:00.621229 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:00.606020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.2.81.66:47391)
0924 12:46:00.606827 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:00.606855 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:00.607061 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:00.607471 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:00.607477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:00.608192 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:00.608632 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.608649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.610043 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.610047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.611328 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:00.611341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:00.612212 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:00.612221 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:00.612369 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:00.613183 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:00.613215 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:00.614261 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:00.618148 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.618154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.618158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.618677 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.619238 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:00.619244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:00.619247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:00.619402 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:00.620048 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:00.620053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:00.620257 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 12:46:00.620394 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 12:46:00.620765 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":404,"threads_started":1}
I20260924 12:46:00.624261 3844 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41541
I20260924 12:46:00.624562 3844 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:00.625473 3844 heartbeater.cc:511] Master 127.2.81.126:41541 requested a full tablet report, sending...
I20260924 12:46:00.627038 3498 ts_manager.cc:194] Registered new tserver with Master: 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825)
I20260924 12:46:00.627916 3498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:59889
I20260924 12:46:00.629164 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3709
I20260924 12:46:00.629415 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:00.629935 2373 external_mini_cluster.cc:1428] Setting key 4a51b540f4aff4351dbdfcdcbf526b17
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:00.673448 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41541
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:01.007890 3848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:01.008177 3848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:01.008315 3848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:01.015676 3848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:01.015825 3848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:01.015944 3848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:01.016055 3848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:01.026299 3848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:01.027659 3848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:01.029112 3848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:01.042157 3853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:01.042253 3854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:01.042860 3856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:01.042959 3848 server_base.cc:1034] running on GCE node
I20260924 12:46:01.043660 3848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:01.044670 3848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:01.045925 3848 hybrid_clock.cc:648] HybridClock initialized: now 1790253961045853 us; error 41 us; skew 500 ppm
I20260924 12:46:01.046257 3848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:01 dist-test-slave-9tm1 krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:01.049678 3848 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:01.051257 3848 webserver.cc:460] Webserver started at http://127.2.81.67:38519/ using document root <none> and password file <none>
I20260924 12:46:01.052002 3848 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:01.052107 3848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:01.052423 3848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:01.055460 3848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "c9e49b9e42664466a8d5af729f088bd1"
format_stamp: "Formatted at 2026-09-24 12:46:01 on dist-test-slave-9tm1"
server_key: "6dd20484a4167a34669b70f8e0d0289a"
server_key_iv: "c76bb55cb11d9744628ab22372590208"
server_key_version: "encryptionkey@0"
I20260924 12:46:01.056245 3848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "c9e49b9e42664466a8d5af729f088bd1"
format_stamp: "Formatted at 2026-09-24 12:46:01 on dist-test-slave-9tm1"
server_key: "6dd20484a4167a34669b70f8e0d0289a"
server_key_iv: "c76bb55cb11d9744628ab22372590208"
server_key_version: "encryptionkey@0"
I20260924 12:46:01.061697 3848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260924 12:46:01.066074 3863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:01.067797 3848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260924 12:46:01.067981 3848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "c9e49b9e42664466a8d5af729f088bd1"
format_stamp: "Formatted at 2026-09-24 12:46:01 on dist-test-slave-9tm1"
server_key: "6dd20484a4167a34669b70f8e0d0289a"
server_key_iv: "c76bb55cb11d9744628ab22372590208"
server_key_version: "encryptionkey@0"
I20260924 12:46:01.068197 3848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:01.090531 3705 heartbeater.cc:503] Master 127.2.81.126:41541 was elected leader, sending a full tablet report...
I20260924 12:46:01.090782 3848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:01.095367 3848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:01.095721 3848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:01.099046 3848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:01.099149 3848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:01.099290 3848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:01.099409 3848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 12:46:01 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:01.123224 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.099484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:57309 (local address 127.2.81.126:41541)
0924 12:46:01.099680 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.099686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.101229 (+ 1543us) server_negotiation.cc:407] Connection header received
0924 12:46:01.103306 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.103313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.103414 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.103545 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:01.105929 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.107170 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.109740 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.110424 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.110586 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.115330 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:01.115338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:01.115343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:01.115392 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:01.119709 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.120604 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.120611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.120616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.120782 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.121363 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.121368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.121372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.121746 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:01.121894 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.122622 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.122854 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 12:46:01.124307 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.100079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.2.81.67:57309)
0924 12:46:01.101034 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.101090 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.103008 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.103904 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.103916 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.104691 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:01.105684 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.105711 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.107404 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.107410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.109542 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.109558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.110727 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.110735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.110992 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.111814 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:01.111840 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:01.114970 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:01.119916 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:01.119927 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:01.119972 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:01.120475 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:01.120906 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:01.120911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:01.120915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:01.121241 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:01.121871 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:01.121880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:01.122248 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.122489 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.123035 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":413,"threads_started":1}
I20260924 12:46:01.143718 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.130033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:58029 (local address 127.2.81.126:41541)
0924 12:46:01.130287 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.130294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.130793 (+ 499us) server_negotiation.cc:407] Connection header received
0924 12:46:01.130952 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.130958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.131047 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.131193 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:01.132165 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.133408 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.135388 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.135917 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.136036 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.138176 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:01.138183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:01.138187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:01.138220 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:01.140682 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.141492 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.141496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.141498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.141625 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.142093 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.142098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.142102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.142466 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:01.142602 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.143133 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.143354 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:01.143910 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.130055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.2.81.67:58029)
0924 12:46:01.130686 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.130706 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.130841 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.131215 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.131218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.131671 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:01.131997 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.132007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.133592 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.133598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.135185 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.135208 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.136137 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.136144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.136301 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.136884 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:01.136901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:01.137984 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:01.140825 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:01.140831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:01.140835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:01.141324 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:01.141785 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:01.141790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:01.141794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:01.141977 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:01.142566 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:01.142570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:01.142803 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.143030 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.143429 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":359,"threads_started":1}
I20260924 12:46:01.152940 3848 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:44449
I20260924 12:46:01.153021 3982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:44449 every 8 connection(s)
I20260924 12:46:01.154129 3848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:01.156189 3983 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41541
I20260924 12:46:01.156486 3983 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:01.157205 3983 heartbeater.cc:511] Master 127.2.81.126:41541 requested a full tablet report, sending...
I20260924 12:46:01.158601 3498 ts_manager.cc:194] Registered new tserver with Master: c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449)
I20260924 12:46:01.159363 3498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:57309
I20260924 12:46:01.162019 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 3848
I20260924 12:46:01.162173 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:01.162519 2373 external_mini_cluster.cc:1428] Setting key 47f82eae8e3c501e4cb15ad2cafa02b0
I20260924 12:46:01.165431 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:46:01 dist-test-slave-9tm1 krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:46:01 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:01.210387 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.195712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.2.81.126:41541)
0924 12:46:01.195976 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.195983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.196269 (+ 286us) server_negotiation.cc:407] Connection header received
0924 12:46:01.196743 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.196747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.196874 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.197001 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:01.198448 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.199177 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.200475 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.200951 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.201099 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.205089 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:01.205095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:01.205098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:01.205136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:01.207713 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.208495 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.208499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.208502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.208634 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.209031 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.209036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.209039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.209319 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:01.209569 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.209835 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.210044 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:46:01.214660 3498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:01.221896 3498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:01.243875 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.234558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.2.81.66:33825)
0924 12:46:01.235045 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.235054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.235864 (+ 810us) server_negotiation.cc:407] Connection header received
0924 12:46:01.236610 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.236616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.236872 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.237078 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.238081 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.239352 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.241541 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.242751 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.242867 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.243070 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.243446 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.243606 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":366,"threads_started":1}
I20260924 12:46:01.244050 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.234551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:33825 (local address 127.0.0.1:49312)
0924 12:46:01.235710 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.235751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.236430 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.237348 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.237355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.237394 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.237910 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.237926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.239493 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.239498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.241388 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.241402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.242962 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.242968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.243154 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.243268 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.243386 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.243575 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":128,"spinlock_wait_cycles":1408,"thread_start_us":633,"threads_started":1}
I20260924 12:46:01.245370 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.235468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.2.81.67:44449)
0924 12:46:01.236465 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.236477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.236511 (+ 34us) server_negotiation.cc:407] Connection header received
0924 12:46:01.236608 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.236616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.236872 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.237090 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.238903 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.240344 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.242715 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.244124 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.244237 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.244440 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.244909 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.245066 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":1444,"threads_started":1}
I20260924 12:46:01.245379 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.234907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:44449 (local address 127.0.0.1:35354)
0924 12:46:01.236060 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.236077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.236234 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.237348 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.237355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.237402 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.238393 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.238405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.240471 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.240476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.242532 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.242545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.244560 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.244566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.244711 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.244741 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.244840 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.244963 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":602,"spinlock_wait_cycles":8320,"thread_start_us":750,"threads_started":1}
I20260924 12:46:01.248180 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.236006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:46845 (local address 127.0.0.1:51742)
0924 12:46:01.237888 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.237905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.238032 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.240055 (+ 2023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.240064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.240096 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.240541 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.240558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.242741 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.242746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.245719 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.245732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.247435 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.247441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.247607 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.247638 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.247729 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.247870 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":1671,"threads_started":1}
I20260924 12:46:01.248240 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.236439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.2.81.65:46845)
0924 12:46:01.237796 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.237804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.238007 (+ 203us) server_negotiation.cc:407] Connection header received
0924 12:46:01.239585 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.239593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.239854 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.240037 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.240755 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.242582 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.245864 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.246986 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.247108 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.247304 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.247812 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.247977 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":1080,"threads_started":1}
I20260924 12:46:01.247937 3774 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae011f9f81940d48e8a943d3beadbdd (DEFAULT_TABLE table=test-table [id=11fc3bf801294e4aa9be6b069bf5d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:01.249555 3774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae011f9f81940d48e8a943d3beadbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.249065 3913 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae011f9f81940d48e8a943d3beadbdd (DEFAULT_TABLE table=test-table [id=11fc3bf801294e4aa9be6b069bf5d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:01.250609 3913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae011f9f81940d48e8a943d3beadbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.252157 3635 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae011f9f81940d48e8a943d3beadbdd (DEFAULT_TABLE table=test-table [id=11fc3bf801294e4aa9be6b069bf5d1eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:01.253770 3635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae011f9f81940d48e8a943d3beadbdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.260856 4002 tablet_bootstrap.cc:492] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Bootstrap starting.
I20260924 12:46:01.263924 4002 tablet_bootstrap.cc:654] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.264791 4003 tablet_bootstrap.cc:492] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Bootstrap starting.
I20260924 12:46:01.265264 4002 log.cc:824] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:01.268334 4003 tablet_bootstrap.cc:654] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.269091 4002 tablet_bootstrap.cc:492] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: No bootstrap required, opened a new log
I20260924 12:46:01.269233 4005 tablet_bootstrap.cc:492] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Bootstrap starting.
I20260924 12:46:01.269532 4002 ts_tablet_manager.cc:1397] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260924 12:46:01.269722 4003 log.cc:824] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:01.272670 4005 tablet_bootstrap.cc:654] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.273415 4003 tablet_bootstrap.cc:492] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: No bootstrap required, opened a new log
I20260924 12:46:01.273833 4003 ts_tablet_manager.cc:1397] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:46:01.273942 4005 log.cc:824] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:01.278697 4005 tablet_bootstrap.cc:492] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: No bootstrap required, opened a new log
I20260924 12:46:01.279018 4005 ts_tablet_manager.cc:1397] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260924 12:46:01.281306 4002 raft_consensus.cc:348] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.281841 4002 raft_consensus.cc:374] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.281980 4002 raft_consensus.cc:729] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515e40103de444ee8fd9e6ac50e7586c, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.282680 4002 consensus_queue.cc:260] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.284456 3844 heartbeater.cc:503] Master 127.2.81.126:41541 was elected leader, sending a full tablet report...
I20260924 12:46:01.285283 4003 raft_consensus.cc:348] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.285877 4003 raft_consensus.cc:374] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.286051 4003 raft_consensus.cc:729] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e49b9e42664466a8d5af729f088bd1, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.286124 4002 ts_tablet_manager.cc:1428] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260924 12:46:01.286921 4003 consensus_queue.cc:260] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.288710 3983 heartbeater.cc:503] Master 127.2.81.126:41541 was elected leader, sending a full tablet report...
I20260924 12:46:01.288578 4005 raft_consensus.cc:348] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.288905 4003 ts_tablet_manager.cc:1428] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 12:46:01.289080 4005 raft_consensus.cc:374] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.289179 4005 raft_consensus.cc:729] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f021935cf998440882bc67ad94bb0bf2, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.290144 4005 consensus_queue.cc:260] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.292038 4005 ts_tablet_manager.cc:1428] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
W20260924 12:46:01.331594 3706 tablet.cc:2369] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:01.374431 4010 raft_consensus.cc:482] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:01.374817 4010 raft_consensus.cc:504] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
W20260924 12:46:01.375308 3845 tablet.cc:2369] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:01.376799 4010 leader_election.cc:288] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825), c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449)
I20260924 12:46:01.383263 4009 raft_consensus.cc:482] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:01.383672 4009 raft_consensus.cc:504] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.384500 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.377393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:40867 (local address 127.2.81.67:44449)
0924 12:46:01.377572 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.377577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.378738 (+ 1161us) server_negotiation.cc:407] Connection header received
0924 12:46:01.378917 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.378921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.378978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.379049 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.379597 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.380294 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.382264 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.383118 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.383234 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.383363 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.384041 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.384181 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:46:01.384519 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.377795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:33825 (local address 127.2.81.65:50203)
0924 12:46:01.378728 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.378743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.378850 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.379150 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.379158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.379211 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.379564 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.379574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.380688 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.380692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.382233 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.382246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.383536 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.383542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.383741 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.383777 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.383871 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.384009 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":3,"spinlock_wait_cycles":7040,"thread_start_us":476,"threads_started":1}
I20260924 12:46:01.384955 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.377248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:50203 (local address 127.2.81.66:33825)
0924 12:46:01.377500 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.377506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.378812 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 12:46:01.378991 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.378995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.379051 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.379123 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.379710 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.380534 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.382407 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.383223 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.383356 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.383482 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.383924 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.384065 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:46:01.385247 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.377767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:44449 (local address 127.2.81.65:40867)
0924 12:46:01.378621 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.378637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.378791 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.379076 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.379080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.379108 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.379449 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.379462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.380429 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.380434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.382114 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.382126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.383692 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.383697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.383869 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.383905 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.384772 (+ 867us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.384899 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":72,"thread_start_us":465,"threads_started":1}
I20260924 12:46:01.385916 3798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae011f9f81940d48e8a943d3beadbdd" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c" is_pre_election: true
I20260924 12:46:01.386175 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae011f9f81940d48e8a943d3beadbdd" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e49b9e42664466a8d5af729f088bd1" is_pre_election: true
I20260924 12:46:01.386483 3798 raft_consensus.cc:2454] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 0.
I20260924 12:46:01.386615 3938 raft_consensus.cc:2454] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 0.
I20260924 12:46:01.387270 4009 leader_election.cc:288] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825), f021935cf998440882bc67ad94bb0bf2 (127.2.81.65:46845)
I20260924 12:46:01.387326 3588 leader_election.cc:302] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e49b9e42664466a8d5af729f088bd1, f021935cf998440882bc67ad94bb0bf2; no voters:
I20260924 12:46:01.387737 4010 raft_consensus.cc:2781] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:01.387914 4010 raft_consensus.cc:482] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:01.388029 4010 raft_consensus.cc:3037] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.391134 4010 raft_consensus.cc:504] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.392009 4010 leader_election.cc:288] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825), c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449)
I20260924 12:46:01.392329 3798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae011f9f81940d48e8a943d3beadbdd" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c"
I20260924 12:46:01.392539 3798 raft_consensus.cc:3037] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.394484 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.386556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:33825 (local address 127.2.81.67:57027)
0924 12:46:01.386842 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.386861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.386988 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.388320 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.388325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.388354 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.388560 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.388573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.390149 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.390156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.391944 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.391969 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.393481 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.393487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.393803 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.393840 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.393949 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.394117 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":5}
I20260924 12:46:01.395102 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.386723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:57027 (local address 127.2.81.66:33825)
0924 12:46:01.387508 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.387514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.387531 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:46:01.387589 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.387594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.387674 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.388699 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.388828 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.389978 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.392157 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.393312 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.393430 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.393543 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.394760 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.394879 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621}
I20260924 12:46:01.394585 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae011f9f81940d48e8a943d3beadbdd" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e49b9e42664466a8d5af729f088bd1"
I20260924 12:46:01.395473 3938 raft_consensus.cc:3037] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.396076 3797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae011f9f81940d48e8a943d3beadbdd" candidate_uuid: "c9e49b9e42664466a8d5af729f088bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c" is_pre_election: true
I20260924 12:46:01.396422 3798 raft_consensus.cc:2454] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 1.
I20260924 12:46:01.397423 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.388147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:60123 (local address 127.2.81.65:46845)
0924 12:46:01.388393 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.388399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.390639 (+ 2240us) server_negotiation.cc:407] Connection header received
0924 12:46:01.390814 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.390818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.390891 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.390984 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:01.391687 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.392791 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.395056 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.396146 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.396278 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.396372 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.397062 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.397199 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 12:46:01.397531 3586 leader_election.cc:302] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515e40103de444ee8fd9e6ac50e7586c, f021935cf998440882bc67ad94bb0bf2; no voters:
I20260924 12:46:01.397972 4010 raft_consensus.cc:2781] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:01.398027 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.387812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:46845 (local address 127.2.81.67:60123)
0924 12:46:01.390527 (+ 2715us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.390546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.390671 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.391028 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.391033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.391064 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:01.391504 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.391516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.393002 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.393008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.394887 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.394911 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.396527 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.396535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.396722 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.396755 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.397567 (+ 812us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.397687 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2566,"thread_start_us":2415,"threads_started":1}
I20260924 12:46:01.399068 4010 raft_consensus.cc:686] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 1 LEADER]: Becoming Leader. State: Replica: f021935cf998440882bc67ad94bb0bf2, State: Running, Role: LEADER
I20260924 12:46:01.399348 3938 raft_consensus.cc:2454] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 1.
I20260924 12:46:01.399461 3655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae011f9f81940d48e8a943d3beadbdd" candidate_uuid: "c9e49b9e42664466a8d5af729f088bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f021935cf998440882bc67ad94bb0bf2" is_pre_election: true
I20260924 12:46:01.399699 4010 consensus_queue.cc:237] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.400640 3865 leader_election.cc:302] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9e49b9e42664466a8d5af729f088bd1; no voters: 515e40103de444ee8fd9e6ac50e7586c, f021935cf998440882bc67ad94bb0bf2
I20260924 12:46:01.401067 4009 raft_consensus.cc:2726] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:46:01.404773 3496 catalog_manager.cc:5495] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f021935cf998440882bc67ad94bb0bf2 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } health_report { overall_health: UNKNOWN } } }
W20260924 12:46:01.405658 3984 tablet.cc:2369] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:01.454473 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.448697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.2.81.65:46845)
0924 12:46:01.448900 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.448907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.449057 (+ 150us) server_negotiation.cc:407] Connection header received
0924 12:46:01.449385 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.449390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.449486 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.449611 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:01.450014 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.450739 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.452153 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.452555 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.452670 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.453260 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:01.453816 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:01.453951 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.454078 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.454162 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:46:01.463207 3798 raft_consensus.cc:1261] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Refusing update from remote peer f021935cf998440882bc67ad94bb0bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:01.463364 3938 raft_consensus.cc:1261] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Refusing update from remote peer f021935cf998440882bc67ad94bb0bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:01.464494 4014 consensus_queue.cc:1035] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:01.464855 4010 consensus_queue.cc:1035] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:01.480091 4017 mvcc.cc:205] Tried to move back new op lower bound from 7332880226142892032 to 7332880225897349120. Current Snapshot: MvccSnapshot[applied={T|T < 7332880226142892032}]
I20260924 12:46:01.482347 4019 mvcc.cc:205] Tried to move back new op lower bound from 7332880226142892032 to 7332880225897349120. Current Snapshot: MvccSnapshot[applied={T|T < 7332880226142892032}]
I20260924 12:46:01.501302 3496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:01.501610 3496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:01.505553 3496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: Sending DeleteTablet for 3 replicas of tablet 7ae011f9f81940d48e8a943d3beadbdd
I20260924 12:46:01.506834 3635 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae011f9f81940d48e8a943d3beadbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:01 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:51742
I20260924 12:46:01.506827 3774 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae011f9f81940d48e8a943d3beadbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:01 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:49312
I20260924 12:46:01.506875 3913 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae011f9f81940d48e8a943d3beadbdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:01 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:35354
I20260924 12:46:01.509443 3496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:01.510438 3496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:01.520206 3635 tablet_service.cc:1467] Processing CreateTablet for tablet b577338181624513bd848b8d6853467b (DEFAULT_TABLE table=test-table [id=c5175e68ca764d83b9bd64fdd13663ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:01.520200 3912 tablet_service.cc:1467] Processing CreateTablet for tablet b577338181624513bd848b8d6853467b (DEFAULT_TABLE table=test-table [id=c5175e68ca764d83b9bd64fdd13663ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:01.520200 3773 tablet_service.cc:1467] Processing CreateTablet for tablet b577338181624513bd848b8d6853467b (DEFAULT_TABLE table=test-table [id=c5175e68ca764d83b9bd64fdd13663ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:01.520737 3773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b577338181624513bd848b8d6853467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.520752 3912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b577338181624513bd848b8d6853467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.520901 3635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b577338181624513bd848b8d6853467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.527112 4002 tablet_bootstrap.cc:492] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Bootstrap starting.
I20260924 12:46:01.527361 4005 tablet_bootstrap.cc:492] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Bootstrap starting.
I20260924 12:46:01.527446 4003 tablet_bootstrap.cc:492] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Bootstrap starting.
I20260924 12:46:01.528752 4002 tablet_bootstrap.cc:654] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.528863 4003 tablet_bootstrap.cc:654] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.529170 4005 tablet_bootstrap.cc:654] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.536420 4027 tablet_replica.cc:330] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: stopping tablet replica
I20260924 12:46:01.536958 4027 raft_consensus.cc:2229] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:01.537698 4027 raft_consensus.cc:2258] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:01.538297 4026 tablet_replica.cc:330] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: stopping tablet replica
I20260924 12:46:01.538769 4003 tablet_bootstrap.cc:492] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: No bootstrap required, opened a new log
I20260924 12:46:01.538816 4026 raft_consensus.cc:2229] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:01.538932 4003 ts_tablet_manager.cc:1397] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20260924 12:46:01.539615 4026 raft_consensus.cc:2258] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:01.539736 4003 raft_consensus.cc:348] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.539973 4003 raft_consensus.cc:374] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.540050 4003 raft_consensus.cc:729] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e49b9e42664466a8d5af729f088bd1, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.540326 4003 consensus_queue.cc:260] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.541802 4003 ts_tablet_manager.cc:1428] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:01.543081 4002 tablet_bootstrap.cc:492] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: No bootstrap required, opened a new log
I20260924 12:46:01.543260 4002 ts_tablet_manager.cc:1397] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.008s
I20260924 12:46:01.544081 4002 raft_consensus.cc:348] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.544436 4002 raft_consensus.cc:374] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.544579 4002 raft_consensus.cc:729] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515e40103de444ee8fd9e6ac50e7586c, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.544891 4002 consensus_queue.cc:260] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.546744 4002 ts_tablet_manager.cc:1428] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:01.548202 4028 tablet_replica.cc:330] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: stopping tablet replica
I20260924 12:46:01.548713 4028 raft_consensus.cc:2229] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:01.550175 4028 raft_consensus.cc:2258] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:01.550273 4005 tablet_bootstrap.cc:492] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: No bootstrap required, opened a new log
I20260924 12:46:01.550486 4005 ts_tablet_manager.cc:1397] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.014s
I20260924 12:46:01.551229 4026 ts_tablet_manager.cc:1905] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:01.551762 4005 raft_consensus.cc:348] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.552021 4005 raft_consensus.cc:374] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.552131 4005 raft_consensus.cc:729] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f021935cf998440882bc67ad94bb0bf2, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.552428 4005 consensus_queue.cc:260] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.551191 4027 ts_tablet_manager.cc:1905] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:01.554687 4028 ts_tablet_manager.cc:1905] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:01.560499 4026 ts_tablet_manager.cc:1918] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:01.560827 4026 log.cc:1196] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/7ae011f9f81940d48e8a943d3beadbdd
I20260924 12:46:01.561633 4026 ts_tablet_manager.cc:1939] T 7ae011f9f81940d48e8a943d3beadbdd P f021935cf998440882bc67ad94bb0bf2: Deleting consensus metadata
I20260924 12:46:01.562520 4027 ts_tablet_manager.cc:1918] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:01.563095 4027 log.cc:1196] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/7ae011f9f81940d48e8a943d3beadbdd
I20260924 12:46:01.563668 4027 ts_tablet_manager.cc:1939] T 7ae011f9f81940d48e8a943d3beadbdd P 515e40103de444ee8fd9e6ac50e7586c: Deleting consensus metadata
I20260924 12:46:01.564075 4028 ts_tablet_manager.cc:1918] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:01.564246 4028 log.cc:1196] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/7ae011f9f81940d48e8a943d3beadbdd
I20260924 12:46:01.564774 4028 ts_tablet_manager.cc:1939] T 7ae011f9f81940d48e8a943d3beadbdd P c9e49b9e42664466a8d5af729f088bd1: Deleting consensus metadata
I20260924 12:46:01.565621 3483 catalog_manager.cc:4841] TS 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825): tablet 7ae011f9f81940d48e8a943d3beadbdd (table test-table [id=11fc3bf801294e4aa9be6b069bf5d1eb]) successfully deleted
I20260924 12:46:01.566643 4005 ts_tablet_manager.cc:1428] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20260924 12:46:01.567911 3483 catalog_manager.cc:4841] TS f021935cf998440882bc67ad94bb0bf2 (127.2.81.65:46845): tablet 7ae011f9f81940d48e8a943d3beadbdd (table test-table [id=11fc3bf801294e4aa9be6b069bf5d1eb]) successfully deleted
I20260924 12:46:01.568290 3485 catalog_manager.cc:4841] TS c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449): tablet 7ae011f9f81940d48e8a943d3beadbdd (table test-table [id=11fc3bf801294e4aa9be6b069bf5d1eb]) successfully deleted
I20260924 12:46:01.568660 4009 raft_consensus.cc:482] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:01.568809 4009 raft_consensus.cc:504] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.569494 4009 leader_election.cc:288] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f021935cf998440882bc67ad94bb0bf2 (127.2.81.65:46845), 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825)
I20260924 12:46:01.570170 3655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b577338181624513bd848b8d6853467b" candidate_uuid: "c9e49b9e42664466a8d5af729f088bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f021935cf998440882bc67ad94bb0bf2" is_pre_election: true
I20260924 12:46:01.570181 3798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b577338181624513bd848b8d6853467b" candidate_uuid: "c9e49b9e42664466a8d5af729f088bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c" is_pre_election: true
I20260924 12:46:01.570413 3798 raft_consensus.cc:2454] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e49b9e42664466a8d5af729f088bd1 in term 0.
I20260924 12:46:01.570473 3655 raft_consensus.cc:2454] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e49b9e42664466a8d5af729f088bd1 in term 0.
I20260924 12:46:01.570880 3865 leader_election.cc:302] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e49b9e42664466a8d5af729f088bd1, f021935cf998440882bc67ad94bb0bf2; no voters:
I20260924 12:46:01.571122 4009 raft_consensus.cc:2781] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:01.571260 4009 raft_consensus.cc:482] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:01.571367 4009 raft_consensus.cc:3037] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.574750 4009 raft_consensus.cc:504] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.575340 4009 leader_election.cc:288] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [CANDIDATE]: Term 1 election: Requested vote from peers f021935cf998440882bc67ad94bb0bf2 (127.2.81.65:46845), 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825)
I20260924 12:46:01.575973 3798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b577338181624513bd848b8d6853467b" candidate_uuid: "c9e49b9e42664466a8d5af729f088bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c"
I20260924 12:46:01.576088 3655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b577338181624513bd848b8d6853467b" candidate_uuid: "c9e49b9e42664466a8d5af729f088bd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f021935cf998440882bc67ad94bb0bf2"
I20260924 12:46:01.576210 3798 raft_consensus.cc:3037] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.576339 3655 raft_consensus.cc:3037] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.579308 3798 raft_consensus.cc:2454] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e49b9e42664466a8d5af729f088bd1 in term 1.
I20260924 12:46:01.579939 3865 leader_election.cc:302] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515e40103de444ee8fd9e6ac50e7586c, c9e49b9e42664466a8d5af729f088bd1; no voters:
I20260924 12:46:01.580477 4009 raft_consensus.cc:2781] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:01.580716 3655 raft_consensus.cc:2454] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e49b9e42664466a8d5af729f088bd1 in term 1.
I20260924 12:46:01.584158 4009 raft_consensus.cc:686] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 1 LEADER]: Becoming Leader. State: Replica: c9e49b9e42664466a8d5af729f088bd1, State: Running, Role: LEADER
I20260924 12:46:01.584910 4009 consensus_queue.cc:237] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.588887 3496 catalog_manager.cc:5495] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67). New cstate: current_term: 1 leader_uuid: "c9e49b9e42664466a8d5af729f088bd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } health_report { overall_health: HEALTHY } } }
Sep 24 12:46:01 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253959, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:01.650148 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.627736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41541 (local address 127.0.0.1:59990)
0924 12:46:01.633604 (+ 5868us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.633630 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.633804 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.634205 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.634210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.635162 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:01.635526 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.635540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.636935 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.636941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.638571 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.638588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.639695 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.639702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.639888 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.640650 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:01.640674 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:01.643640 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:01.647335 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:01.647347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:01.647353 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:01.647942 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:01.648361 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:01.648364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:01.648367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:01.648507 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:01.649236 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:01.649243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:01.649445 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.649591 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.649817 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5678,"thread_start_us":5438,"threads_started":1}
I20260924 12:46:01.650658 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.627620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.2.81.126:41541)
0924 12:46:01.627800 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.627806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.633747 (+ 5941us) server_negotiation.cc:407] Connection header received
0924 12:46:01.633924 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.633930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.634024 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.634131 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:01.635706 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.636771 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.638788 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.639499 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.639625 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.643873 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:01.643880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:01.643885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:01.643924 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:01.647161 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.648084 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.648091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.648095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.648248 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:01.648612 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:01.648617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:01.648620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:01.649084 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:01.649230 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.649768 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.650055 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 12:46:01.657315 3496 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:59990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 12:46:01.658448 3496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:01.665270 3773 tablet_service.cc:1467] Processing CreateTablet for tablet 798cb407b00a418db4de8bd3487e5cee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b58a6c8b3084a09bca3611e547a4aea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:01.665501 3635 tablet_service.cc:1467] Processing CreateTablet for tablet 798cb407b00a418db4de8bd3487e5cee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b58a6c8b3084a09bca3611e547a4aea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:01.665830 3773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798cb407b00a418db4de8bd3487e5cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.665995 3635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798cb407b00a418db4de8bd3487e5cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.669075 3913 tablet_service.cc:1467] Processing CreateTablet for tablet 798cb407b00a418db4de8bd3487e5cee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b58a6c8b3084a09bca3611e547a4aea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:01.669637 3913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798cb407b00a418db4de8bd3487e5cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:01.673224 4005 tablet_bootstrap.cc:492] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: Bootstrap starting.
I20260924 12:46:01.675410 4005 tablet_bootstrap.cc:654] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.677356 4003 tablet_bootstrap.cc:492] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: Bootstrap starting.
I20260924 12:46:01.678831 4003 tablet_bootstrap.cc:654] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.679087 4005 tablet_bootstrap.cc:492] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: No bootstrap required, opened a new log
I20260924 12:46:01.679246 4005 ts_tablet_manager.cc:1397] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 12:46:01.680073 4005 raft_consensus.cc:348] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.680394 4002 tablet_bootstrap.cc:492] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: Bootstrap starting.
I20260924 12:46:01.680523 4003 tablet_bootstrap.cc:492] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: No bootstrap required, opened a new log
I20260924 12:46:01.680661 4003 ts_tablet_manager.cc:1397] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:01.680683 4005 raft_consensus.cc:374] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.680773 4005 raft_consensus.cc:729] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f021935cf998440882bc67ad94bb0bf2, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.681023 4005 consensus_queue.cc:260] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.681384 4003 raft_consensus.cc:348] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.681600 4003 raft_consensus.cc:374] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.681469 4014 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.681667 4003 raft_consensus.cc:729] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e49b9e42664466a8d5af729f088bd1, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.681777 4014 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.681989 4002 tablet_bootstrap.cc:654] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:01.681926 4003 consensus_queue.cc:260] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.682297 4009 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.682536 4009 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.682924 4003 ts_tablet_manager.cc:1428] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:01.684726 4005 ts_tablet_manager.cc:1428] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260924 12:46:01.687073 4002 tablet_bootstrap.cc:492] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: No bootstrap required, opened a new log
I20260924 12:46:01.687242 4002 ts_tablet_manager.cc:1397] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260924 12:46:01.687978 4002 raft_consensus.cc:348] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.688244 4002 raft_consensus.cc:374] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:01.688318 4002 raft_consensus.cc:729] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515e40103de444ee8fd9e6ac50e7586c, State: Initialized, Role: FOLLOWER
I20260924 12:46:01.688613 4002 consensus_queue.cc:260] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.689070 4008 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.689472 4008 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.689962 4002 ts_tablet_manager.cc:1428] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 12:46:01.850999 4014 raft_consensus.cc:482] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:01.851300 4014 raft_consensus.cc:504] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.852131 4014 leader_election.cc:288] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825), c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449)
I20260924 12:46:01.852759 3798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798cb407b00a418db4de8bd3487e5cee" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c" is_pre_election: true
I20260924 12:46:01.853091 3798 raft_consensus.cc:2454] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 0.
I20260924 12:46:01.852741 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798cb407b00a418db4de8bd3487e5cee" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e49b9e42664466a8d5af729f088bd1" is_pre_election: true
I20260924 12:46:01.853523 3938 raft_consensus.cc:2454] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 0.
I20260924 12:46:01.853569 3586 leader_election.cc:302] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515e40103de444ee8fd9e6ac50e7586c, f021935cf998440882bc67ad94bb0bf2; no voters:
I20260924 12:46:01.853893 4014 raft_consensus.cc:2781] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:01.854034 4014 raft_consensus.cc:482] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:01.854168 4014 raft_consensus.cc:3037] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.857427 4014 raft_consensus.cc:504] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.858225 4014 leader_election.cc:288] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825), c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449)
I20260924 12:46:01.858754 3798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798cb407b00a418db4de8bd3487e5cee" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515e40103de444ee8fd9e6ac50e7586c"
I20260924 12:46:01.858834 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798cb407b00a418db4de8bd3487e5cee" candidate_uuid: "f021935cf998440882bc67ad94bb0bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e49b9e42664466a8d5af729f088bd1"
I20260924 12:46:01.858978 3798 raft_consensus.cc:3037] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.859071 3938 raft_consensus.cc:3037] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:01.860898 3798 raft_consensus.cc:2454] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 1.
I20260924 12:46:01.861378 3586 leader_election.cc:302] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515e40103de444ee8fd9e6ac50e7586c, f021935cf998440882bc67ad94bb0bf2; no voters:
I20260924 12:46:01.861632 4014 raft_consensus.cc:2781] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:01.861944 4014 raft_consensus.cc:686] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [term 1 LEADER]: Becoming Leader. State: Replica: f021935cf998440882bc67ad94bb0bf2, State: Running, Role: LEADER
I20260924 12:46:01.862210 3938 raft_consensus.cc:2454] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f021935cf998440882bc67ad94bb0bf2 in term 1.
I20260924 12:46:01.862260 4014 consensus_queue.cc:237] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } }
I20260924 12:46:01.864344 4010 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: TxnStatusTablet state changed. Reason: New leader f021935cf998440882bc67ad94bb0bf2. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.864637 4010 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:01.865330 4044 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 12:46:01.865563 4044 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 12:46:01.865548 3496 catalog_manager.cc:5495] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f021935cf998440882bc67ad94bb0bf2 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:01.928493 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.921759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.2.81.65:46845)
0924 12:46:01.921923 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.921929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.922221 (+ 292us) server_negotiation.cc:407] Connection header received
0924 12:46:01.922562 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.922567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.922657 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.922739 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:01.923153 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.923967 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.926181 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.926838 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.926951 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.927597 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:01.927739 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:01.927863 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.928176 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.928257 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 12:46:01.928521 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.921761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:46845 (local address 127.0.0.1:51774)
0924 12:46:01.922122 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.922143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.922310 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.922800 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.922805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.922829 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:01.923024 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.923036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.924107 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.924112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.925949 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.925970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.927026 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.927033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.927238 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.927327 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:01.927909 (+ 582us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:01.927938 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.928035 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.928203 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260924 12:46:01.933897 3798 raft_consensus.cc:1261] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Refusing update from remote peer f021935cf998440882bc67ad94bb0bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:01.933933 3938 raft_consensus.cc:1261] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Refusing update from remote peer f021935cf998440882bc67ad94bb0bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:01.935401 4010 consensus_queue.cc:1035] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:01.935761 4014 consensus_queue.cc:1035] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:01.939543 4008 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: TxnStatusTablet state changed. Reason: New leader f021935cf998440882bc67ad94bb0bf2. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.939867 4008 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.940539 4009 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: TxnStatusTablet state changed. Reason: New leader f021935cf998440882bc67ad94bb0bf2. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.940841 4009 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.943331 4018 mvcc.cc:205] Tried to move back new op lower bound from 7332880228069548032 to 7332880227790516224. Current Snapshot: MvccSnapshot[applied={T|T < 7332880228069548032}]
I20260924 12:46:01.943789 4009 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.944059 4009 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P c9e49b9e42664466a8d5af729f088bd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.949173 4010 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.949530 4010 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:01.951045 4014 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.951182 4008 tablet_replica.cc:439] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f021935cf998440882bc67ad94bb0bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 } } peers { permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 } } peers { permanent_uuid: "c9e49b9e42664466a8d5af729f088bd1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44449 } } }
I20260924 12:46:01.951458 4008 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P 515e40103de444ee8fd9e6ac50e7586c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:01.952147 4014 tablet_replica.cc:442] T 798cb407b00a418db4de8bd3487e5cee P f021935cf998440882bc67ad94bb0bf2: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:01.968854 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.959162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.2.81.67:44449)
0924 12:46:01.961591 (+ 2429us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.961596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.961681 (+ 85us) server_negotiation.cc:407] Connection header received
0924 12:46:01.961801 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.961805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.961917 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.962113 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:01.963509 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.964779 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.966497 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.966969 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.967109 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.967585 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:01.968171 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:01.968314 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.968437 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.968540 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2300,"thread_start_us":2177,"threads_started":1}
I20260924 12:46:01.990018 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.982488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:46845 (local address 127.2.81.67:46369)
0924 12:46:01.983177 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:01.983196 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:01.983364 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:01.983752 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:01.983757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:01.983771 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:01.984094 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.984107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.985818 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.985840 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.987558 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:01.987578 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.988426 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.988432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:01.988611 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.988691 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:01.989223 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:01.989236 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 12:46:01.989325 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 12:46:01.989498 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":338,"threads_started":1}
I20260924 12:46:01.990145 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:01.982612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:46369 (local address 127.2.81.65:46845)
0924 12:46:01.982840 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:01.982847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:01.983343 (+ 496us) server_negotiation.cc:407] Connection header received
0924 12:46:01.983563 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:01.983567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:01.983636 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:01.983714 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:01.984313 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.985579 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.987753 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:01.988237 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:01.988354 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:01.988911 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:01.989126 (+ 215us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:01.989200 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:01.989398 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 12:46:01.989474 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:02.002082 3798 raft_consensus.cc:1261] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Refusing update from remote peer c9e49b9e42664466a8d5af729f088bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:02.002506 3655 raft_consensus.cc:1261] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Refusing update from remote peer c9e49b9e42664466a8d5af729f088bd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:02.002908 4032 consensus_queue.cc:1035] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515e40103de444ee8fd9e6ac50e7586c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:02.003568 4009 consensus_queue.cc:1035] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f021935cf998440882bc67ad94bb0bf2" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:02.047364 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.039306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:35391 (local address 127.2.81.67:44449)
0924 12:46:02.039974 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:02.039981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:02.040431 (+ 450us) server_negotiation.cc:407] Connection header received
0924 12:46:02.040866 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:02.040872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:02.040961 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:02.041093 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:02.041739 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.042870 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.045109 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.045799 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.045915 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.046644 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:02.046761 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:02.046838 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:02.046974 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 12:46:02.047066 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531}
I20260924 12:46:02.047528 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.039287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:44449 (local address 127.2.81.65:35391)
0924 12:46:02.040321 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:02.040342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:02.040551 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:02.041116 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:02.041121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:02.041154 (+ 33us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:02.041536 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:02.041551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.043063 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.043068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:02.044913 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:02.044928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.046112 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.046119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:02.046312 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.046412 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:02.046866 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:02.046877 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 12:46:02.046962 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 12:46:02.047121 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":683,"threads_started":1}
I20260924 12:46:02.089830 3496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:02.090066 3496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:02.093804 3496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b9c31785f144f1bbbceb22588812e1c: Sending DeleteTablet for 3 replicas of tablet b577338181624513bd848b8d6853467b
I20260924 12:46:02.094877 3773 tablet_service.cc:1514] Processing DeleteTablet for tablet b577338181624513bd848b8d6853467b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:02 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:49312
I20260924 12:46:02.095796 3635 tablet_service.cc:1514] Processing DeleteTablet for tablet b577338181624513bd848b8d6853467b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:02 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:51742
I20260924 12:46:02.096208 3913 tablet_service.cc:1514] Processing DeleteTablet for tablet b577338181624513bd848b8d6853467b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:02 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:35354
I20260924 12:46:02.107971 4065 tablet_replica.cc:330] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: stopping tablet replica
I20260924 12:46:02.109077 4066 tablet_replica.cc:330] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: stopping tablet replica
I20260924 12:46:02.113338 4065 raft_consensus.cc:2229] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:02.114835 4065 raft_consensus.cc:2258] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:02.115459 4064 tablet_replica.cc:330] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: stopping tablet replica
I20260924 12:46:02.115828 4064 raft_consensus.cc:2229] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:02.116122 4064 raft_consensus.cc:2258] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:02.116724 4065 ts_tablet_manager.cc:1905] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:02.119629 4064 ts_tablet_manager.cc:1905] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:02.125102 4066 raft_consensus.cc:2229] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:02.125504 4065 ts_tablet_manager.cc:1918] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:02.125708 4065 log.cc:1196] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/b577338181624513bd848b8d6853467b
I20260924 12:46:02.125792 4066 raft_consensus.cc:2258] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:02.126247 4065 ts_tablet_manager.cc:1939] T b577338181624513bd848b8d6853467b P f021935cf998440882bc67ad94bb0bf2: Deleting consensus metadata
I20260924 12:46:02.127513 3483 catalog_manager.cc:4841] TS f021935cf998440882bc67ad94bb0bf2 (127.2.81.65:46845): tablet b577338181624513bd848b8d6853467b (table test-table [id=c5175e68ca764d83b9bd64fdd13663ec]) successfully deleted
I20260924 12:46:02.127897 4066 ts_tablet_manager.cc:1905] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:02.132556 4064 ts_tablet_manager.cc:1918] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:02.137827 4066 ts_tablet_manager.cc:1918] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:02.138450 4066 log.cc:1196] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/b577338181624513bd848b8d6853467b
I20260924 12:46:02.139073 4066 ts_tablet_manager.cc:1939] T b577338181624513bd848b8d6853467b P c9e49b9e42664466a8d5af729f088bd1: Deleting consensus metadata
I20260924 12:46:02.141155 3485 catalog_manager.cc:4841] TS c9e49b9e42664466a8d5af729f088bd1 (127.2.81.67:44449): tablet b577338181624513bd848b8d6853467b (table test-table [id=c5175e68ca764d83b9bd64fdd13663ec]) successfully deleted
I20260924 12:46:02.143913 4064 log.cc:1196] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/b577338181624513bd848b8d6853467b
I20260924 12:46:02.145102 4064 ts_tablet_manager.cc:1939] T b577338181624513bd848b8d6853467b P 515e40103de444ee8fd9e6ac50e7586c: Deleting consensus metadata
I20260924 12:46:02.146462 3483 catalog_manager.cc:4841] TS 515e40103de444ee8fd9e6ac50e7586c (127.2.81.66:33825): tablet b577338181624513bd848b8d6853467b (table test-table [id=c5175e68ca764d83b9bd64fdd13663ec]) successfully deleted
Sep 24 12:46:02 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:02.164623 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.115255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.2.81.65:46845)
0924 12:46:02.115433 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:02.115439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:02.134592 (+ 19153us) server_negotiation.cc:407] Connection header received
0924 12:46:02.141415 (+ 6823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:02.141422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:02.141514 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:02.141596 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:02.143876 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.145326 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.148245 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.149030 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.149193 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.155099 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:02.155118 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:02.155138 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:02.155200 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:02.160259 (+ 5059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.161442 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.161454 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.161458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.161636 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.162096 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.162101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.162105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.162563 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:02.162876 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:02.163971 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0924 12:46:02.164241 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 12:46:02.166623 3699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51790
I20260924 12:46:02.216044 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.180657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.2.81.126:41541)
0924 12:46:02.199729 (+ 19072us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:02.199736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:02.199761 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:46:02.199827 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:02.199833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:02.199923 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:02.200055 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:02.202973 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.204394 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.206138 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.206573 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.206704 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.209303 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:02.209309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:02.209313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:02.209355 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:02.212752 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.213835 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.213847 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.213851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.214028 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.214462 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.214466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.214469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.214758 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:02.215027 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:02.215369 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 12:46:02.215626 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18799,"thread_start_us":18957,"threads_started":1}
W20260924 12:46:02.216714 3496 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60002
I20260924 12:46:02.219426 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3569
I20260924 12:46:02.236390 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.222083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.2.81.65:46845)
0924 12:46:02.222301 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:02.222308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:02.223072 (+ 764us) server_negotiation.cc:407] Connection header received
0924 12:46:02.223244 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:02.223251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:02.223326 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:02.223410 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:02.224593 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.225485 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.226842 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.227357 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.227498 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.229844 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:02.229849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:02.229853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:02.229896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:02.233285 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.234312 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.234318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.234322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.234528 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.234892 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.234898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.234902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.235196 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:02.235471 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:02.235739 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 12:46:02.235988 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 12:46:02.237178 3699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51794
W20260924 12:46:02.237824 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:02.238049 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3569
I20260924 12:46:02.260921 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3709
Sep 24 12:46:02 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:46:02.280087 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.261725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.2.81.66:33825)
0924 12:46:02.262233 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:02.262237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:02.262255 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:02.262310 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:02.262313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:02.262380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:02.262473 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:02.264122 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.264903 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.267043 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.267649 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.267793 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.273346 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:02.273363 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:02.273380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:02.273428 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:02.276644 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.277813 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.277824 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.277828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.278049 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.278493 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.278499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.278503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.278885 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:02.279175 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:02.279499 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 12:46:02.279739 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
W20260924 12:46:02.281129 3839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49320
W20260924 12:46:02.281773 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3709: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:02.281952 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3709
I20260924 12:46:02.303022 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3848
Sep 24 12:46:02 dist-test-slave-9tm1 krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:46:02.321907 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:02.303760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.2.81.67:44449)
0924 12:46:02.303947 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:02.303951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:02.303969 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:02.304181 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:02.304186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:02.304252 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:02.304358 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:02.306198 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.307024 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.308866 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:02.309442 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:02.309555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:02.315173 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:02.315192 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:02.315209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:02.315251 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:02.318577 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.319453 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.319461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.319463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.319597 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:02.320143 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:02.320150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:02.320153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:02.320660 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:02.320923 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:02.321363 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 12:46:02.321572 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 12:46:02.322711 3978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35364
W20260924 12:46:02.323414 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:02.323581 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3848
I20260924 12:46:02.339515 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 3466
I20260924 12:46:02.340728 3538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:02.458033 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 3466
2026-09-24T12:46:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4642 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:02 dist-test-slave-9tm1 krb5kdc[4077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:02 dist-test-slave-9tm1 krb5kdc[4077](info): set up 2 sockets
Sep 24 12:46:02 dist-test-slave-9tm1 krb5kdc[4077](info): commencing operation
krb5kdc: starting...
W20260924 12:46:03.489310 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:03 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:03.685931 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:35197
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:35197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:04.015738 4090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:04.016057 4090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:04.016201 4090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:04.023692 4090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:04.023834 4090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 12:46:04.023907 4090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:04.024003 4090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:04.024076 4090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:04.024158 4090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:04.034143 4090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:35197
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:04.035426 4090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:04.036695 4090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:04.048650 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:04.048774 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:04.048739 4098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:04.049919 4090 server_base.cc:1034] running on GCE node
I20260924 12:46:04.050784 4090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:04.052338 4090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:04.053586 4090 hybrid_clock.cc:648] HybridClock initialized: now 1790253964053518 us; error 47 us; skew 500 ppm
I20260924 12:46:04.053933 4090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:04 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253964, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:04.057235 4090 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:04.058822 4090 webserver.cc:460] Webserver started at http://127.2.81.126:36077/ using document root <none> and password file <none>
I20260924 12:46:04.059535 4090 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:04.059634 4090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:04.059940 4090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:04.062928 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "aa8a88eede424f6cb346c6d8b0eb97b8"
format_stamp: "Formatted at 2026-09-24 12:46:04 on dist-test-slave-9tm1"
server_key: "b7f6ae3c8ce57270cf9cae505d22deaa"
server_key_iv: "d22504abddbab43d4a15946b71267c9a"
server_key_version: "encryptionkey@0"
I20260924 12:46:04.063673 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "aa8a88eede424f6cb346c6d8b0eb97b8"
format_stamp: "Formatted at 2026-09-24 12:46:04 on dist-test-slave-9tm1"
server_key: "b7f6ae3c8ce57270cf9cae505d22deaa"
server_key_iv: "d22504abddbab43d4a15946b71267c9a"
server_key_version: "encryptionkey@0"
I20260924 12:46:04.068857 4090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:46:04.072594 4105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:04.074154 4090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:46:04.074334 4090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "aa8a88eede424f6cb346c6d8b0eb97b8"
format_stamp: "Formatted at 2026-09-24 12:46:04 on dist-test-slave-9tm1"
server_key: "b7f6ae3c8ce57270cf9cae505d22deaa"
server_key_iv: "d22504abddbab43d4a15946b71267c9a"
server_key_version: "encryptionkey@0"
I20260924 12:46:04.074544 4090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:04.093240 4090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:04.097857 4090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:04.098131 4090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:04.118800 4090 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:35197
I20260924 12:46:04.118819 4166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:35197 every 8 connection(s)
I20260924 12:46:04.120177 4090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:04.123600 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 4090
I20260924 12:46:04.123816 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:04.124307 4167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:04.124591 2373 external_mini_cluster.cc:1428] Setting key 9ddc8416a6cf585ae5b6847a7708f480
I20260924 12:46:04.138918 4167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Bootstrap starting.
Sep 24 12:46:04 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:04.142100 4167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:04.143148 4167 log.cc:824] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:04.145988 4167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: No bootstrap required, opened a new log
I20260924 12:46:04.147128 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:04.126650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.2.81.126:35197)
0924 12:46:04.127435 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:04.127452 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:04.127493 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:46:04.129163 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:04.129187 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:04.129611 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:04.129991 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:04.131774 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.133139 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.134781 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.135544 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.135706 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:04.140162 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:04.140170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:04.140185 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:04.140230 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:04.143168 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:04.143877 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:04.143884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:04.143891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:04.144070 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:04.144426 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:04.144430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:04.144432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:04.144953 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:04.145249 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:04.145672 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 12:46:04.146093 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":376,"threads_started":1}
I20260924 12:46:04.153950 4167 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35197 } }
I20260924 12:46:04.154330 4167 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:04.154449 4167 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8a88eede424f6cb346c6d8b0eb97b8, State: Initialized, Role: FOLLOWER
I20260924 12:46:04.154963 4167 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35197 } }
I20260924 12:46:04.155164 4167 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:04.155256 4167 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:04.155392 4167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:04.157778 4167 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35197 } }
I20260924 12:46:04.158308 4167 leader_election.cc:302] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8a88eede424f6cb346c6d8b0eb97b8; no voters:
I20260924 12:46:04.158874 4167 leader_election.cc:288] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:04.159036 4172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:04.159655 4172 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [term 1 LEADER]: Becoming Leader. State: Replica: aa8a88eede424f6cb346c6d8b0eb97b8, State: Running, Role: LEADER
I20260924 12:46:04.160179 4172 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35197 } }
I20260924 12:46:04.160801 4167 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:04.162686 4173 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35197 } } }
I20260924 12:46:04.162804 4174 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8a88eede424f6cb346c6d8b0eb97b8. Latest consensus state: current_term: 1 leader_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8a88eede424f6cb346c6d8b0eb97b8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35197 } } }
I20260924 12:46:04.163123 4174 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:04.163232 4173 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:04.165828 4181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:04.171247 4181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:04.181556 4181 catalog_manager.cc:1347] Generated new cluster ID: 01183a8cf0d44a03a1a37b6f3b9e2f11
I20260924 12:46:04.181696 4181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:04.208536 4181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:04.209937 4181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:04.217756 4181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Generated new TSK 0
I20260924 12:46:04.218609 4181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:46:04.282902 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35197
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:04.667902 4193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:04.668212 4193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:04.668315 4193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:04.676141 4193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:04.676282 4193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:04.676380 4193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 12:46:04.676434 4193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:04.676496 4193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:04.686350 4193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:35197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:04.687583 4193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:04.688875 4193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:04.703717 4198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:04.703755 4199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:04.703776 4201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:04.705312 4193 server_base.cc:1034] running on GCE node
I20260924 12:46:04.706007 4193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:04.707031 4193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:04.708279 4193 hybrid_clock.cc:648] HybridClock initialized: now 1790253964708192 us; error 82 us; skew 500 ppm
I20260924 12:46:04.708643 4193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:04 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253964, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:04.711874 4193 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:46:04.713455 4193 webserver.cc:460] Webserver started at http://127.2.81.65:35659/ using document root <none> and password file <none>
I20260924 12:46:04.714154 4193 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:04.714283 4193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:04.714583 4193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:04.717542 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "6d765d74811f4955a4f0fbaafb2325e8"
format_stamp: "Formatted at 2026-09-24 12:46:04 on dist-test-slave-9tm1"
server_key: "89721e9e0a66024e9c884ffe4f298cf9"
server_key_iv: "1b5beae7166069a16a03f3d5eff85c6e"
server_key_version: "encryptionkey@0"
I20260924 12:46:04.718287 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "6d765d74811f4955a4f0fbaafb2325e8"
format_stamp: "Formatted at 2026-09-24 12:46:04 on dist-test-slave-9tm1"
server_key: "89721e9e0a66024e9c884ffe4f298cf9"
server_key_iv: "1b5beae7166069a16a03f3d5eff85c6e"
server_key_version: "encryptionkey@0"
I20260924 12:46:04.723502 4193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 12:46:04.727183 4208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:04.728700 4193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 12:46:04.728879 4193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "6d765d74811f4955a4f0fbaafb2325e8"
format_stamp: "Formatted at 2026-09-24 12:46:04 on dist-test-slave-9tm1"
server_key: "89721e9e0a66024e9c884ffe4f298cf9"
server_key_iv: "1b5beae7166069a16a03f3d5eff85c6e"
server_key_version: "encryptionkey@0"
I20260924 12:46:04.729126 4193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:04.762621 4193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:04.767385 4193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:04.767669 4193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:04.770303 4193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:04.770407 4193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:04.770526 4193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:04.770599 4193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:04 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253964, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:04.792759 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:04.770824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:58285 (local address 127.2.81.126:35197)
0924 12:46:04.771456 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:04.771462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:04.772612 (+ 1150us) server_negotiation.cc:407] Connection header received
0924 12:46:04.774870 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:04.774877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:04.774977 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:04.775056 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:04.777729 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.778643 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.780293 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.780684 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.780794 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:04.785169 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:04.785175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:04.785178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:04.785214 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:04.788489 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:04.789706 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:04.789712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:04.789715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:04.789886 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:04.790576 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:04.790581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:04.790584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:04.790948 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:04.791117 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:04.792307 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0924 12:46:04.792499 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":390,"threads_started":1}
I20260924 12:46:04.794683 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:04.771567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.2.81.65:58285)
0924 12:46:04.772427 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:04.772477 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:04.774531 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:04.775708 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:04.775717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:04.776540 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:04.777438 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:04.777471 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.778806 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.778809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:04.780138 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:04.780152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.780918 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.780925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:04.781201 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:04.782055 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:04.782079 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:04.784887 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:04.788686 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:04.788698 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:04.788725 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:04.789503 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:04.790033 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:04.790038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:04.790043 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:04.790400 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:04.791091 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:04.791100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:04.791757 (+ 657us) client_negotiation.cc:769] Sending connection context
0924 12:46:04.792199 (+ 442us) client_negotiation.cc:240] Negotiation successful
0924 12:46:04.792949 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":361,"threads_started":1}
I20260924 12:46:04.815653 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:04.800960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.2.81.65:34273)
0924 12:46:04.801696 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:04.801712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:04.801834 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:04.802162 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:04.802165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:04.802629 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:04.802916 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:04.802926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.804412 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.804417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:04.805724 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:04.805741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.806730 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.806744 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:04.806966 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:04.807894 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:04.807921 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:04.809086 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:04.812779 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:04.812785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:04.812790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:04.813273 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:04.813661 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:04.813666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:04.813670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:04.813797 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:04.814371 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:04.814376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:04.814626 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 12:46:04.814804 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 12:46:04.815131 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":447,"threads_started":1}
I20260924 12:46:04.816423 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:04.800968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:34273 (local address 127.2.81.126:35197)
0924 12:46:04.801186 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:04.801192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:04.801792 (+ 600us) server_negotiation.cc:407] Connection header received
0924 12:46:04.801938 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:04.801944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:04.802035 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:04.802149 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:04.803054 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.804223 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.805905 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:04.806518 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:04.806659 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:04.809342 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:04.809348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:04.809351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:04.809387 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:04.812611 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:04.813406 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:04.813412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:04.813416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:04.813552 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:04.813909 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:04.813914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:04.813918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:04.814267 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:04.814408 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:04.816013 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0924 12:46:04.816190 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:46:04.824376 4193 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:42555
I20260924 12:46:04.824463 4328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:42555 every 8 connection(s)
I20260924 12:46:04.825637 4193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:04.826841 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 4193
I20260924 12:46:04.827020 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:04.827696 2373 external_mini_cluster.cc:1428] Setting key a35834b4204c2864b6a265d46503a6d3
I20260924 12:46:04.827813 4329 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35197
I20260924 12:46:04.828060 4329 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:04.828763 4329 heartbeater.cc:511] Master 127.2.81.126:35197 requested a full tablet report, sending...
I20260924 12:46:04.830938 4122 ts_manager.cc:194] Registered new tserver with Master: 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555)
I20260924 12:46:04.832962 4122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:58285
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:04.869602 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35197
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:05.200455 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:05.200760 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:05.200889 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:05.208262 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:05.208608 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:05.208860 4333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 12:46:05.208954 4333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:05.209111 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:05.220263 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:35197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:05.221791 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:05.223100 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:05.236086 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:05.236115 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:05.236054 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:05.236789 4333 server_base.cc:1034] running on GCE node
I20260924 12:46:05.237448 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:05.238420 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:05.239650 4333 hybrid_clock.cc:648] HybridClock initialized: now 1790253965239588 us; error 57 us; skew 500 ppm
I20260924 12:46:05.239995 4333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:05 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253965, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:05.243223 4333 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:05.244907 4333 webserver.cc:460] Webserver started at http://127.2.81.66:45849/ using document root <none> and password file <none>
I20260924 12:46:05.245661 4333 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:05.245769 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:05.246049 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:05.249184 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "8adcc7f1432d4438843d1b2f9d7dd289"
format_stamp: "Formatted at 2026-09-24 12:46:05 on dist-test-slave-9tm1"
server_key: "f89c241408561cd444ecb1122b39f106"
server_key_iv: "9023d66400481d59e42a712a49b29263"
server_key_version: "encryptionkey@0"
I20260924 12:46:05.250017 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "8adcc7f1432d4438843d1b2f9d7dd289"
format_stamp: "Formatted at 2026-09-24 12:46:05 on dist-test-slave-9tm1"
server_key: "f89c241408561cd444ecb1122b39f106"
server_key_iv: "9023d66400481d59e42a712a49b29263"
server_key_version: "encryptionkey@0"
I20260924 12:46:05.258147 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260924 12:46:05.262392 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:05.263947 4333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:46:05.264173 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "8adcc7f1432d4438843d1b2f9d7dd289"
format_stamp: "Formatted at 2026-09-24 12:46:05 on dist-test-slave-9tm1"
server_key: "f89c241408561cd444ecb1122b39f106"
server_key_iv: "9023d66400481d59e42a712a49b29263"
server_key_version: "encryptionkey@0"
I20260924 12:46:05.264384 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:05.310207 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:05.314797 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:05.315091 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:05.318049 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:05.318176 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:05.318295 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:05.318391 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:05 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253965, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:05.342362 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.318562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:40343 (local address 127.2.81.126:35197)
0924 12:46:05.319367 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:05.319372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:05.320352 (+ 980us) server_negotiation.cc:407] Connection header received
0924 12:46:05.322592 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:05.322598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:05.322696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:05.322790 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:05.325285 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.326468 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.329315 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.329919 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.330052 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.334784 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:05.334791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:05.334796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:05.334841 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:05.338118 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.339264 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.339270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.339274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.339471 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.340183 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.340189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.340193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.340543 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:05.340683 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:05.341875 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0924 12:46:05.342081 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":616,"threads_started":1}
I20260924 12:46:05.344092 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.319294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.2.81.66:40343)
0924 12:46:05.320164 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:05.320211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:05.322254 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:05.323096 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:05.323112 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:05.324076 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:05.325006 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.325039 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.326808 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.326818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.329139 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.329157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.330186 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.330194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.330542 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.331415 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:05.331452 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:05.334484 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:05.338343 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.338358 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.338379 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.339069 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.339577 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.339583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.339587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.339917 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.340706 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:05.340717 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:05.341357 (+ 640us) client_negotiation.cc:769] Sending connection context
0924 12:46:05.341740 (+ 383us) client_negotiation.cc:240] Negotiation successful
0924 12:46:05.342406 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":385,"threads_started":1}
I20260924 12:46:05.368455 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.351035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:59363 (local address 127.2.81.126:35197)
0924 12:46:05.351258 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:05.351265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:05.354375 (+ 3110us) server_negotiation.cc:407] Connection header received
0924 12:46:05.354982 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:05.354987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:05.355062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:05.355149 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:05.356128 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.357095 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.358466 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.358883 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.359007 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.361650 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:05.361655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:05.361659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:05.361694 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:05.364959 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.365991 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.365998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.366001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.366179 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.366631 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.366637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.366641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.367139 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:05.367316 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:05.367825 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 12:46:05.368091 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:46:05.368604 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.350940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.2.81.66:59363)
0924 12:46:05.354207 (+ 3267us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:05.354233 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:05.354503 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:05.355206 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:05.355211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:05.355672 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:05.355968 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.355980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.357251 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.357255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.358312 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.358325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.359087 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.359093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.359290 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.360047 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:05.360076 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:05.361417 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:05.365149 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.365155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.365159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.365846 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.366314 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.366320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.366324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.366490 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.367283 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:05.367288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:05.367521 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 12:46:05.367725 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 12:46:05.368128 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3007,"thread_start_us":2854,"threads_started":1}
I20260924 12:46:05.371788 4333 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:46765
I20260924 12:46:05.371825 4468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:46765 every 8 connection(s)
I20260924 12:46:05.373136 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:05.375248 4469 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35197
I20260924 12:46:05.375490 4469 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:05.376168 4469 heartbeater.cc:511] Master 127.2.81.126:35197 requested a full tablet report, sending...
I20260924 12:46:05.377547 4122 ts_manager.cc:194] Registered new tserver with Master: 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765)
I20260924 12:46:05.378506 4122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:40343
I20260924 12:46:05.379319 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 4333
I20260924 12:46:05.379568 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:05.380035 2373 external_mini_cluster.cc:1428] Setting key d2b60e3e227c36fe6ec69b380113db2c
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:05.422086 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35197
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:05.762261 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:05.762588 4473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:05.762738 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:05.771473 4473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:05.771636 4473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:05.771771 4473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 12:46:05.771838 4473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:05.771951 4473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:05.782589 4473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:35197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:05.784061 4473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:05.785526 4473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:05.800159 4478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:05.800129 4481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:05.800196 4479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:05.801191 4473 server_base.cc:1034] running on GCE node
I20260924 12:46:05.801860 4473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:05.803483 4473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:05.804768 4473 hybrid_clock.cc:648] HybridClock initialized: now 1790253965804724 us; error 108 us; skew 500 ppm
I20260924 12:46:05.805223 4473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:05 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253965, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:05.814163 4473 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:05.815871 4473 webserver.cc:460] Webserver started at http://127.2.81.67:45079/ using document root <none> and password file <none>
I20260924 12:46:05.816589 4473 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:05.816690 4473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:05.816955 4473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:05.819979 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
format_stamp: "Formatted at 2026-09-24 12:46:05 on dist-test-slave-9tm1"
server_key: "76b169017a14777b9554961ccf5c58bd"
server_key_iv: "9a7fb284b1404860cc3869c3c1351f1d"
server_key_version: "encryptionkey@0"
I20260924 12:46:05.820709 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
format_stamp: "Formatted at 2026-09-24 12:46:05 on dist-test-slave-9tm1"
server_key: "76b169017a14777b9554961ccf5c58bd"
server_key_iv: "9a7fb284b1404860cc3869c3c1351f1d"
server_key_version: "encryptionkey@0"
I20260924 12:46:05.825997 4473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:46:05.829412 4488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:05.830827 4473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:46:05.830982 4473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
format_stamp: "Formatted at 2026-09-24 12:46:05 on dist-test-slave-9tm1"
server_key: "76b169017a14777b9554961ccf5c58bd"
server_key_iv: "9a7fb284b1404860cc3869c3c1351f1d"
server_key_version: "encryptionkey@0"
I20260924 12:46:05.831185 4473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:05.836447 4329 heartbeater.cc:503] Master 127.2.81.126:35197 was elected leader, sending a full tablet report...
I20260924 12:46:05.849735 4473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:05.854532 4473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:05.854842 4473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:05.858001 4473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:05.858117 4473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:05.858237 4473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:05.858316 4473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:05 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253965, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:05.881640 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.859204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:33411 (local address 127.2.81.126:35197)
0924 12:46:05.859434 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:05.859441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:05.860339 (+ 898us) server_negotiation.cc:407] Connection header received
0924 12:46:05.862637 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:05.862643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:05.862728 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:05.862812 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:05.865188 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.866394 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.869044 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.869541 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.869720 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.874717 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:05.874725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:05.874728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:05.874762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:05.877351 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.878641 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.878647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.878650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.878822 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.879466 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.879471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.879475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.879850 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:05.879986 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:05.881046 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 12:46:05.881278 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 12:46:05.882686 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.859191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.2.81.67:33411)
0924 12:46:05.860133 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:05.860185 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:05.862292 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:05.863107 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:05.863117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:05.864001 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:05.864842 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.864870 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.866609 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.866616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.868865 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.868883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.869867 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.869876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.870226 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.870987 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:05.871020 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:05.874315 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:05.877561 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.877574 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.877599 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.878476 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.878965 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.878969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.878975 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.879312 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.880123 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:05.880132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:05.880515 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 12:46:05.880912 (+ 397us) client_negotiation.cc:240] Negotiation successful
0924 12:46:05.881510 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":377,"threads_started":1}
I20260924 12:46:05.909518 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.895311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.2.81.67:57769)
0924 12:46:05.895901 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:05.895921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:05.896077 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:05.896397 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:05.896401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:05.896948 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:05.897367 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.897383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.898494 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.898498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.899494 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:05.899507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.900371 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.900379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:05.900574 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.901381 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:05.901408 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:05.902641 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:05.906597 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.906603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.906610 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.907213 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.907689 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:05.907693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:05.907695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:05.907832 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:05.908536 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:05.908540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:05.908730 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 12:46:05.908852 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 12:46:05.909184 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":330,"threads_started":1}
I20260924 12:46:05.909622 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.895303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:57769 (local address 127.2.81.126:35197)
0924 12:46:05.895507 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:05.895511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:05.896014 (+ 503us) server_negotiation.cc:407] Connection header received
0924 12:46:05.896192 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:05.896198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:05.896267 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:05.896355 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:05.897528 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.898339 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.899631 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.900124 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.900246 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.902908 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:05.902915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:05.902918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:05.902951 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:05.906429 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.907349 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.907355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.907359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.907548 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.907975 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.907982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.907986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.908385 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:05.908552 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:05.908967 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 12:46:05.909243 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 12:46:05.917330 4473 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:33615
I20260924 12:46:05.917374 4607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:33615 every 8 connection(s)
I20260924 12:46:05.918582 4473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:05.920521 4608 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35197
I20260924 12:46:05.920717 4608 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:05.921089 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 4473
I20260924 12:46:05.921240 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:05.921478 4608 heartbeater.cc:511] Master 127.2.81.126:35197 requested a full tablet report, sending...
I20260924 12:46:05.921670 2373 external_mini_cluster.cc:1428] Setting key 5c9b432b503e5d51bf7ebc36e5767297
I20260924 12:46:05.922923 4121 ts_manager.cc:194] Registered new tserver with Master: 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:05.923873 4121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:33411
I20260924 12:46:05.924510 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:46:05 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:46:05 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:05.969818 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.955536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.2.81.126:35197)
0924 12:46:05.955754 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:05.955760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:05.955854 (+ 94us) server_negotiation.cc:407] Connection header received
0924 12:46:05.956045 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:05.956049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:05.956111 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:05.956193 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:05.957577 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.958299 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.959874 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:05.960299 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:05.960418 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:05.964550 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:05.964555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:05.964558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:05.964599 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:05.967180 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.967872 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.967877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.967881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.968013 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:05.968389 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:05.968395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:05.968399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:05.968676 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:05.968928 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:05.969246 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 12:46:05.969500 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:46:05.973970 4121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:05.980792 4121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:06.029211 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.993406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.2.81.65:42555)
0924 12:46:06.020952 (+ 27546us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.020959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.021020 (+ 61us) server_negotiation.cc:407] Connection header received
0924 12:46:06.021162 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.021171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.021470 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.022278 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.022549 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.023642 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.026681 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.027743 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.027860 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.028060 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.028720 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.028892 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27344,"thread_start_us":27224,"threads_started":1}
I20260924 12:46:06.029675 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.992953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42555 (local address 127.0.0.1:58090)
0924 12:46:06.020882 (+ 27929us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.020901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.021035 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.021719 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.021729 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.021760 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.022339 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.022352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.023769 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.023774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.026470 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.026485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.028192 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.028199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.028403 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.028434 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.028545 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.029285 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27806,"mutex_wait_us":2362,"thread_start_us":24367,"threads_started":1}
I20260924 12:46:06.029785 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.993475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.2.81.67:33615)
0924 12:46:06.020719 (+ 27244us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.020727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.020759 (+ 32us) server_negotiation.cc:407] Connection header received
0924 12:46:06.021164 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.021172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.021449 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.021670 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.022772 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.024384 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.027134 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.028457 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.028572 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.028766 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.029322 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.029486 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27080,"thread_start_us":26995,"threads_started":1}
I20260924 12:46:06.029838 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.993082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:33615 (local address 127.0.0.1:36768)
0924 12:46:06.019102 (+ 26020us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.019185 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.020889 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.021976 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.021980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.022001 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.022600 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.022610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.024523 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.024528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.026962 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.026975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.028921 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.028929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.029134 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.029163 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.029253 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.029381 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25772,"mutex_wait_us":77,"thread_start_us":26935,"threads_started":1}
I20260924 12:46:06.030182 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.994575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.2.81.66:46765)
0924 12:46:06.018444 (+ 23869us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.018456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.020445 (+ 1989us) server_negotiation.cc:407] Connection header received
0924 12:46:06.021333 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.021341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.021592 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.021783 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.022885 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.024357 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.026574 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.027703 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.027840 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.028051 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.029755 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.029931 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23643,"thread_start_us":23500,"threads_started":1}
I20260924 12:46:06.030465 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:05.992953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.0.0.1:58028)
0924 12:46:06.019102 (+ 26149us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.020312 (+ 1210us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.021155 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.022005 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.022014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.022037 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.022678 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.022692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.024523 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.024528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.026412 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.026436 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.027894 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.027899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.028345 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.028411 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.028542 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.028756 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25787,"mutex_wait_us":76,"spinlock_wait_cycles":1523968,"thread_start_us":26933,"threads_started":1}
I20260924 12:46:06.033824 4399 tablet_service.cc:1467] Processing CreateTablet for tablet 42689ac03dc44338aa88b51bf4ec3d15 (DEFAULT_TABLE table=test-table [id=7f8d48bd50fb465082aec44dd316376f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:06.033838 4259 tablet_service.cc:1467] Processing CreateTablet for tablet 42689ac03dc44338aa88b51bf4ec3d15 (DEFAULT_TABLE table=test-table [id=7f8d48bd50fb465082aec44dd316376f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:06.035171 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42689ac03dc44338aa88b51bf4ec3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.035181 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42689ac03dc44338aa88b51bf4ec3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.035533 4538 tablet_service.cc:1467] Processing CreateTablet for tablet 42689ac03dc44338aa88b51bf4ec3d15 (DEFAULT_TABLE table=test-table [id=7f8d48bd50fb465082aec44dd316376f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:06.037014 4538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42689ac03dc44338aa88b51bf4ec3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.047521 4627 tablet_bootstrap.cc:492] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Bootstrap starting.
I20260924 12:46:06.049968 4628 tablet_bootstrap.cc:492] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Bootstrap starting.
I20260924 12:46:06.051653 4627 tablet_bootstrap.cc:654] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.053771 4629 tablet_bootstrap.cc:492] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Bootstrap starting.
I20260924 12:46:06.053851 4628 tablet_bootstrap.cc:654] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.053794 4627 log.cc:824] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:06.055281 4628 log.cc:824] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:06.056962 4629 tablet_bootstrap.cc:654] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.057920 4629 log.cc:824] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:06.063602 4627 tablet_bootstrap.cc:492] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: No bootstrap required, opened a new log
I20260924 12:46:06.063921 4627 ts_tablet_manager.cc:1397] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260924 12:46:06.064050 4629 tablet_bootstrap.cc:492] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: No bootstrap required, opened a new log
I20260924 12:46:06.064304 4629 ts_tablet_manager.cc:1397] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 12:46:06.065147 4628 tablet_bootstrap.cc:492] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: No bootstrap required, opened a new log
I20260924 12:46:06.065409 4628 ts_tablet_manager.cc:1397] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260924 12:46:06.071887 4629 raft_consensus.cc:348] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.072427 4629 raft_consensus.cc:374] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.072547 4629 raft_consensus.cc:729] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b7a9eaefe4497da5d8a38e5b14ac73, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.073216 4629 consensus_queue.cc:260] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.074265 4608 heartbeater.cc:503] Master 127.2.81.126:35197 was elected leader, sending a full tablet report...
I20260924 12:46:06.074950 4629 ts_tablet_manager.cc:1428] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 12:46:06.074720 4627 raft_consensus.cc:348] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.075134 4627 raft_consensus.cc:374] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.075215 4627 raft_consensus.cc:729] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.075785 4628 raft_consensus.cc:348] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.075940 4627 consensus_queue.cc:260] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.076200 4628 raft_consensus.cc:374] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.076333 4628 raft_consensus.cc:729] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.077039 4628 consensus_queue.cc:260] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.077217 4469 heartbeater.cc:503] Master 127.2.81.126:35197 was elected leader, sending a full tablet report...
I20260924 12:46:06.077929 4627 ts_tablet_manager.cc:1428] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 12:46:06.079387 4628 ts_tablet_manager.cc:1428] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260924 12:46:06.125097 4470 tablet.cc:2369] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:06.170121 4609 tablet.cc:2369] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:06.239518 4634 raft_consensus.cc:482] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:06.239799 4634 raft_consensus.cc:504] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.241489 4634 leader_election.cc:288] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615), 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555)
I20260924 12:46:06.248301 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.241964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:42501 (local address 127.2.81.67:33615)
0924 12:46:06.242193 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.242201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.242893 (+ 692us) server_negotiation.cc:407] Connection header received
0924 12:46:06.243076 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.243080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.243136 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.243209 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.243759 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.244541 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.246398 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.247406 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.247523 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.247630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.247922 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.248059 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:46:06.248396 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.241769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:33615 (local address 127.2.81.66:42501)
0924 12:46:06.242774 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.242791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.242931 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.243237 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.243241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.243263 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.243624 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.243633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.244682 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.244686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.246247 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.246262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.247630 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.247637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.247774 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.247798 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.247882 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.248020 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":5,"thread_start_us":626,"threads_started":1}
I20260924 12:46:06.249498 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42689ac03dc44338aa88b51bf4ec3d15" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" is_pre_election: true
I20260924 12:46:06.249969 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.242011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:51349 (local address 127.2.81.65:42555)
0924 12:46:06.242200 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.242206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.242976 (+ 770us) server_negotiation.cc:407] Connection header received
0924 12:46:06.243169 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.243173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.243242 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.243318 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.244069 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.245213 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.247018 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.248784 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.248918 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.249046 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.249563 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.249715 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:46:06.250094 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.242029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42555 (local address 127.2.81.66:51349)
0924 12:46:06.242863 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.242880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.243024 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.243349 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.243353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.243392 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.243874 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.243886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.245399 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.245403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.246863 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.246876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.249063 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.249069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.249234 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.249259 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.249528 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.249676 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":17,"thread_start_us":441,"threads_started":1}
I20260924 12:46:06.250020 4560 raft_consensus.cc:2454] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 0.
I20260924 12:46:06.250736 4349 leader_election.cc:302] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:06.251204 4634 raft_consensus.cc:2781] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:06.251168 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42689ac03dc44338aa88b51bf4ec3d15" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d765d74811f4955a4f0fbaafb2325e8" is_pre_election: true
I20260924 12:46:06.251359 4634 raft_consensus.cc:482] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:06.251669 4284 raft_consensus.cc:2454] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 0.
I20260924 12:46:06.251637 4634 raft_consensus.cc:3037] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.254489 4634 raft_consensus.cc:504] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.255133 4634 leader_election.cc:288] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 election: Requested vote from peers 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615), 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555)
I20260924 12:46:06.255748 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42689ac03dc44338aa88b51bf4ec3d15" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d765d74811f4955a4f0fbaafb2325e8"
I20260924 12:46:06.255719 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42689ac03dc44338aa88b51bf4ec3d15" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
I20260924 12:46:06.255941 4284 raft_consensus.cc:3037] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.256119 4560 raft_consensus.cc:3037] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.258387 4560 raft_consensus.cc:2454] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 1.
I20260924 12:46:06.258786 4284 raft_consensus.cc:2454] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 1.
I20260924 12:46:06.258985 4349 leader_election.cc:302] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:06.259420 4634 raft_consensus.cc:2781] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:06.259943 4634 raft_consensus.cc:686] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 LEADER]: Becoming Leader. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Running, Role: LEADER
I20260924 12:46:06.260694 4634 consensus_queue.cc:237] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } }
I20260924 12:46:06.267388 4121 catalog_manager.cc:5495] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66). New cstate: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } health_report { overall_health: HEALTHY } } }
W20260924 12:46:06.328511 4330 tablet.cc:2369] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 12:46:06 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253964, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:06.354117 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.339551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.2.81.126:35197)
0924 12:46:06.339748 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.339754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.340262 (+ 508us) server_negotiation.cc:407] Connection header received
0924 12:46:06.340421 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.340424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.340496 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.340576 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:06.341777 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.342469 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.343709 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.344101 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.344208 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.347767 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:06.347771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:06.347774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:06.347806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:06.350988 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:06.351802 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:06.351806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:06.351809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:06.351969 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:06.352355 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:06.352360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:06.352363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:06.352637 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:06.352811 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.353319 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.353640 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":58}
I20260924 12:46:06.354133 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.339502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35197 (local address 127.0.0.1:40224)
0924 12:46:06.340157 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.340173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.340316 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.340619 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.340623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.341357 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:06.341634 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.341646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.342595 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.342599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.343572 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.343586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.344276 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.344282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.344434 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.345166 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:06.345189 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:06.347567 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:06.351156 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:06.351174 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:06.351181 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:06.351679 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:06.352078 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:06.352083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:06.352087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:06.352259 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:06.352819 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:06.352828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:06.353039 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.353189 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.353504 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":341,"threads_started":1}
I20260924 12:46:06.360090 4121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:40224:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 12:46:06.361290 4121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:06.368669 4399 tablet_service.cc:1467] Processing CreateTablet for tablet 1774b7bf68b1405194e7f68e1ece40f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63bf4266c74b41c6b0ebe59fe00c4513]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:06.368736 4259 tablet_service.cc:1467] Processing CreateTablet for tablet 1774b7bf68b1405194e7f68e1ece40f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63bf4266c74b41c6b0ebe59fe00c4513]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:06.368925 4538 tablet_service.cc:1467] Processing CreateTablet for tablet 1774b7bf68b1405194e7f68e1ece40f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63bf4266c74b41c6b0ebe59fe00c4513]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:06.369696 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1774b7bf68b1405194e7f68e1ece40f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.369696 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1774b7bf68b1405194e7f68e1ece40f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.369714 4538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1774b7bf68b1405194e7f68e1ece40f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.376682 4628 tablet_bootstrap.cc:492] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: Bootstrap starting.
I20260924 12:46:06.376674 4629 tablet_bootstrap.cc:492] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: Bootstrap starting.
I20260924 12:46:06.378499 4627 tablet_bootstrap.cc:492] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: Bootstrap starting.
I20260924 12:46:06.378538 4628 tablet_bootstrap.cc:654] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.378669 4629 tablet_bootstrap.cc:654] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.380265 4627 tablet_bootstrap.cc:654] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.387557 4628 tablet_bootstrap.cc:492] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: No bootstrap required, opened a new log
I20260924 12:46:06.387843 4628 ts_tablet_manager.cc:1397] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260924 12:46:06.390244 4628 raft_consensus.cc:348] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.390669 4628 raft_consensus.cc:374] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.390703 4629 tablet_bootstrap.cc:492] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: No bootstrap required, opened a new log
I20260924 12:46:06.390815 4628 raft_consensus.cc:729] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.391099 4629 ts_tablet_manager.cc:1397] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260924 12:46:06.391080 4628 consensus_queue.cc:260] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.391780 4628 ts_tablet_manager.cc:1428] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 12:46:06.391610 4635 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.392012 4635 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.392546 4629 raft_consensus.cc:348] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.392865 4629 raft_consensus.cc:374] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.393015 4629 raft_consensus.cc:729] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b7a9eaefe4497da5d8a38e5b14ac73, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.393427 4627 tablet_bootstrap.cc:492] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: No bootstrap required, opened a new log
I20260924 12:46:06.393318 4629 consensus_queue.cc:260] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.393678 4627 ts_tablet_manager.cc:1397] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260924 12:46:06.393738 4633 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.394007 4633 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.394517 4627 raft_consensus.cc:348] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.394826 4627 raft_consensus.cc:374] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.394981 4627 raft_consensus.cc:729] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.395025 4629 ts_tablet_manager.cc:1428] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 12:46:06.395316 4627 consensus_queue.cc:260] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.395704 4634 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.395991 4634 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.396621 4627 ts_tablet_manager.cc:1428] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:06.448648 4634 raft_consensus.cc:482] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:06.448939 4634 raft_consensus.cc:504] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.449605 4634 leader_election.cc:288] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555), 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:06.450181 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1774b7bf68b1405194e7f68e1ece40f2" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d765d74811f4955a4f0fbaafb2325e8" is_pre_election: true
I20260924 12:46:06.450212 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1774b7bf68b1405194e7f68e1ece40f2" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" is_pre_election: true
I20260924 12:46:06.450407 4284 raft_consensus.cc:2454] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 0.
I20260924 12:46:06.450498 4560 raft_consensus.cc:2454] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 0.
I20260924 12:46:06.450824 4351 leader_election.cc:302] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d765d74811f4955a4f0fbaafb2325e8, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:06.451108 4634 raft_consensus.cc:2781] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:06.451195 4634 raft_consensus.cc:482] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:06.451261 4634 raft_consensus.cc:3037] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.453397 4634 raft_consensus.cc:504] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.454303 4634 leader_election.cc:288] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 election: Requested vote from peers 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555), 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:06.454810 4560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1774b7bf68b1405194e7f68e1ece40f2" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
I20260924 12:46:06.454813 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1774b7bf68b1405194e7f68e1ece40f2" candidate_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d765d74811f4955a4f0fbaafb2325e8"
I20260924 12:46:06.455020 4560 raft_consensus.cc:3037] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.455029 4284 raft_consensus.cc:3037] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.457036 4560 raft_consensus.cc:2454] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 1.
I20260924 12:46:06.457638 4284 raft_consensus.cc:2454] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8adcc7f1432d4438843d1b2f9d7dd289 in term 1.
I20260924 12:46:06.457665 4349 leader_election.cc:302] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:06.457965 4634 raft_consensus.cc:2781] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:06.458195 4634 raft_consensus.cc:686] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 LEADER]: Becoming Leader. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Running, Role: LEADER
I20260924 12:46:06.458467 4634 consensus_queue.cc:237] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.460208 4638 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: TxnStatusTablet state changed. Reason: New leader 8adcc7f1432d4438843d1b2f9d7dd289. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.460619 4638 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:06.461328 4650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 12:46:06.461611 4650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 12:46:06.461953 4122 catalog_manager.cc:5495] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66). New cstate: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:06.496865 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.491549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.2.81.66:46765)
0924 12:46:06.491798 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.491804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.491824 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:46:06.491906 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.491910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.492034 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.492161 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:06.492523 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.493383 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.494617 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.495064 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.495170 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.495657 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.496225 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.496358 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.496468 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.496559 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 12:46:06.496917 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.491211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.0.0.1:58044)
0924 12:46:06.491504 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.491527 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.491665 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.492205 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.492209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.492222 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:06.492390 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.492398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.493507 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.493512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.494478 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.494491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.495219 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.495224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.495388 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.495457 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.496328 (+ 871us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.496358 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.496438 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.496621 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 12:46:06.504921 4284 raft_consensus.cc:1261] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Refusing update from remote peer 8adcc7f1432d4438843d1b2f9d7dd289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:06.504921 4560 raft_consensus.cc:1261] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Refusing update from remote peer 8adcc7f1432d4438843d1b2f9d7dd289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:06.506018 4638 consensus_queue.cc:1035] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:06.506433 4634 consensus_queue.cc:1035] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:06.516633 4635 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: TxnStatusTablet state changed. Reason: New leader 8adcc7f1432d4438843d1b2f9d7dd289. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.516996 4635 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.518625 4633 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: TxnStatusTablet state changed. Reason: New leader 8adcc7f1432d4438843d1b2f9d7dd289. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.518851 4633 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.522192 4634 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.522507 4634 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:06.522749 4634 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.523005 4634 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 8adcc7f1432d4438843d1b2f9d7dd289: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:06.523054 4653 mvcc.cc:205] Tried to move back new op lower bound from 7332880246793732096 to 7332880246616219648. Current Snapshot: MvccSnapshot[applied={T|T < 7332880246793732096}]
I20260924 12:46:06.526166 4635 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.526402 4635 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 6d765d74811f4955a4f0fbaafb2325e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.541294 4633 tablet_replica.cc:439] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } } }
I20260924 12:46:06.541603 4633 tablet_replica.cc:442] T 1774b7bf68b1405194e7f68e1ece40f2 P 18b7a9eaefe4497da5d8a38e5b14ac73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:06.550789 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.540693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.2.81.66:46765)
0924 12:46:06.540914 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.540921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.545625 (+ 4704us) server_negotiation.cc:407] Connection header received
0924 12:46:06.545973 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.545978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.546048 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.546127 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:06.546586 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.547496 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.549004 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.549482 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.549606 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.550048 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.550183 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.550259 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.550412 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.550503 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:46:06.567857 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.562410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:32985 (local address 127.2.81.66:46765)
0924 12:46:06.562989 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.562995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.563170 (+ 175us) server_negotiation.cc:407] Connection header received
0924 12:46:06.563373 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.563376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.563430 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.563499 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:06.563961 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.564723 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.566183 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.566596 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.566699 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.567129 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.567249 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.567320 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.567440 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.567537 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":70}
I20260924 12:46:06.567981 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.562501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.2.81.66:32985)
0924 12:46:06.563074 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.563091 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.563237 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.563518 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.563522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.563531 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:06.563810 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.563818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.564855 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.564860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.566054 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.566068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.566769 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.566775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.566886 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.566946 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.567331 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.567342 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.567418 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.567600 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":1,"thread_start_us":332,"threads_started":1}
I20260924 12:46:06.578706 4559 raft_consensus.cc:1261] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Refusing update from remote peer 8adcc7f1432d4438843d1b2f9d7dd289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:06.578703 4284 raft_consensus.cc:1261] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Refusing update from remote peer 8adcc7f1432d4438843d1b2f9d7dd289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:06.579267 4666 consensus_queue.cc:1035] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:06.579700 4638 consensus_queue.cc:1035] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:06.587607 4654 mvcc.cc:205] Tried to move back new op lower bound from 7332880247098826752 to 7332880245809688576. Current Snapshot: MvccSnapshot[applied={T|T < 7332880247098826752}]
I20260924 12:46:06.595836 4655 mvcc.cc:205] Tried to move back new op lower bound from 7332880247098826752 to 7332880245809688576. Current Snapshot: MvccSnapshot[applied={T|T < 7332880247098826752}]
I20260924 12:46:06.661602 4121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:06.661922 4121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:06.665947 4121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Sending DeleteTablet for 3 replicas of tablet 42689ac03dc44338aa88b51bf4ec3d15
I20260924 12:46:06.667104 4538 tablet_service.cc:1514] Processing DeleteTablet for tablet 42689ac03dc44338aa88b51bf4ec3d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:06 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:36768
I20260924 12:46:06.667307 4398 tablet_service.cc:1514] Processing DeleteTablet for tablet 42689ac03dc44338aa88b51bf4ec3d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:06 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58028
I20260924 12:46:06.667356 4259 tablet_service.cc:1514] Processing DeleteTablet for tablet 42689ac03dc44338aa88b51bf4ec3d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:06 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58090
I20260924 12:46:06.670048 4121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:06.672415 4121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:06.680065 4671 tablet_replica.cc:330] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: stopping tablet replica
I20260924 12:46:06.680745 4671 raft_consensus.cc:2229] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:06.681270 4671 raft_consensus.cc:2258] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:06.683571 4537 tablet_service.cc:1467] Processing CreateTablet for tablet cccd85c001fa4753989610cfa35c008a (DEFAULT_TABLE table=test-table [id=e87c5fe3bff143819c08090d47f79880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:06.683852 4672 tablet_replica.cc:330] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: stopping tablet replica
I20260924 12:46:06.684157 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccd85c001fa4753989610cfa35c008a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.684321 4672 raft_consensus.cc:2229] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:06.685050 4672 raft_consensus.cc:2258] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:06.685881 4670 tablet_replica.cc:330] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: stopping tablet replica
I20260924 12:46:06.686280 4670 raft_consensus.cc:2229] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:06.686650 4670 raft_consensus.cc:2258] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:06.687382 4672 ts_tablet_manager.cc:1905] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:06.688361 4670 ts_tablet_manager.cc:1905] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:06.690565 4399 tablet_service.cc:1467] Processing CreateTablet for tablet cccd85c001fa4753989610cfa35c008a (DEFAULT_TABLE table=test-table [id=e87c5fe3bff143819c08090d47f79880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:06.690997 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccd85c001fa4753989610cfa35c008a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.695390 4670 ts_tablet_manager.cc:1918] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:06.695674 4670 log.cc:1196] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/42689ac03dc44338aa88b51bf4ec3d15
I20260924 12:46:06.696292 4670 ts_tablet_manager.cc:1939] T 42689ac03dc44338aa88b51bf4ec3d15 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting consensus metadata
I20260924 12:46:06.697073 4672 ts_tablet_manager.cc:1918] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:06.697351 4672 log.cc:1196] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/42689ac03dc44338aa88b51bf4ec3d15
I20260924 12:46:06.698103 4672 ts_tablet_manager.cc:1939] T 42689ac03dc44338aa88b51bf4ec3d15 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting consensus metadata
I20260924 12:46:06.698037 4259 tablet_service.cc:1467] Processing CreateTablet for tablet cccd85c001fa4753989610cfa35c008a (DEFAULT_TABLE table=test-table [id=e87c5fe3bff143819c08090d47f79880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:06.698575 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccd85c001fa4753989610cfa35c008a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:06.698518 4106 catalog_manager.cc:4841] TS 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615): tablet 42689ac03dc44338aa88b51bf4ec3d15 (table test-table [id=7f8d48bd50fb465082aec44dd316376f]) successfully deleted
I20260924 12:46:06.699756 4107 catalog_manager.cc:4841] TS 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765): tablet 42689ac03dc44338aa88b51bf4ec3d15 (table test-table [id=7f8d48bd50fb465082aec44dd316376f]) successfully deleted
I20260924 12:46:06.702845 4627 tablet_bootstrap.cc:492] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Bootstrap starting.
I20260924 12:46:06.704523 4627 tablet_bootstrap.cc:654] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.704713 4629 tablet_bootstrap.cc:492] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Bootstrap starting.
I20260924 12:46:06.706210 4671 ts_tablet_manager.cc:1905] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:06.706316 4629 tablet_bootstrap.cc:654] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.713980 4628 tablet_bootstrap.cc:492] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Bootstrap starting.
I20260924 12:46:06.714285 4671 ts_tablet_manager.cc:1918] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:06.714460 4671 log.cc:1196] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/42689ac03dc44338aa88b51bf4ec3d15
I20260924 12:46:06.715081 4671 ts_tablet_manager.cc:1939] T 42689ac03dc44338aa88b51bf4ec3d15 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting consensus metadata
I20260924 12:46:06.715756 4628 tablet_bootstrap.cc:654] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:06.717180 4108 catalog_manager.cc:4841] TS 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555): tablet 42689ac03dc44338aa88b51bf4ec3d15 (table test-table [id=7f8d48bd50fb465082aec44dd316376f]) successfully deleted
I20260924 12:46:06.719410 4628 tablet_bootstrap.cc:492] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: No bootstrap required, opened a new log
I20260924 12:46:06.719420 4629 tablet_bootstrap.cc:492] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: No bootstrap required, opened a new log
I20260924 12:46:06.719584 4629 ts_tablet_manager.cc:1397] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260924 12:46:06.719579 4628 ts_tablet_manager.cc:1397] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260924 12:46:06.719945 4627 tablet_bootstrap.cc:492] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: No bootstrap required, opened a new log
I20260924 12:46:06.720109 4627 ts_tablet_manager.cc:1397] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260924 12:46:06.720275 4628 raft_consensus.cc:348] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.720314 4629 raft_consensus.cc:348] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.720513 4628 raft_consensus.cc:374] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.720537 4629 raft_consensus.cc:374] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.720577 4628 raft_consensus.cc:729] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.720607 4629 raft_consensus.cc:729] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b7a9eaefe4497da5d8a38e5b14ac73, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.720790 4628 consensus_queue.cc:260] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.720820 4629 consensus_queue.cc:260] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.720872 4627 raft_consensus.cc:348] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.721117 4627 raft_consensus.cc:374] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:06.721196 4627 raft_consensus.cc:729] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Initialized, Role: FOLLOWER
I20260924 12:46:06.721374 4628 ts_tablet_manager.cc:1428] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 12:46:06.721539 4629 ts_tablet_manager.cc:1428] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:06.721408 4627 consensus_queue.cc:260] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.723767 4627 ts_tablet_manager.cc:1428] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 12:46:06.744298 4635 raft_consensus.cc:482] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:06.744545 4635 raft_consensus.cc:504] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.746243 4635 leader_election.cc:288] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765), 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:06.754562 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.746641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.2.81.65:46671)
0924 12:46:06.748008 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.748027 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.748217 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.748595 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.748599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.748625 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.749022 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.749039 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.750406 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.750412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.752419 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.752437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.753791 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.753798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.753982 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.754007 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.754094 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.754241 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":674,"threads_started":1}
I20260924 12:46:06.754542 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.747138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:46671 (local address 127.2.81.66:46765)
0924 12:46:06.747325 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.747331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.748110 (+ 779us) server_negotiation.cc:407] Connection header received
0924 12:46:06.748370 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.748375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.748460 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.748563 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.749176 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.750263 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.752623 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.753606 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.753748 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.753844 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.754136 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.754266 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:46:06.755757 4419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccd85c001fa4753989610cfa35c008a" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" is_pre_election: true
I20260924 12:46:06.756232 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.746568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:33615 (local address 127.2.81.65:58559)
0924 12:46:06.747449 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.747471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.747727 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.748124 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.748129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.748155 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:06.748516 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.748531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.750674 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.750684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.753676 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.753696 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.755427 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.755434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.755619 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.755652 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.755756 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.755903 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":1083,"threads_started":1}
I20260924 12:46:06.756239 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.746748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:58559 (local address 127.2.81.67:33615)
0924 12:46:06.746957 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.746964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.747575 (+ 611us) server_negotiation.cc:407] Connection header received
0924 12:46:06.747932 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.747936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.748009 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.748087 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:06.748713 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.749937 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.753911 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.755196 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.755354 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.755463 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.755798 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.755940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:46:06.756186 4419 raft_consensus.cc:2454] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 0.
I20260924 12:46:06.756927 4210 leader_election.cc:302] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d765d74811f4955a4f0fbaafb2325e8, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:06.757055 4559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccd85c001fa4753989610cfa35c008a" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" is_pre_election: true
I20260924 12:46:06.757375 4635 raft_consensus.cc:2781] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:06.757386 4559 raft_consensus.cc:2454] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 0.
I20260924 12:46:06.757472 4635 raft_consensus.cc:482] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:06.757543 4635 raft_consensus.cc:3037] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.759752 4635 raft_consensus.cc:504] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.760468 4635 leader_election.cc:288] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 election: Requested vote from peers 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765), 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:06.761088 4559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccd85c001fa4753989610cfa35c008a" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
I20260924 12:46:06.761087 4419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccd85c001fa4753989610cfa35c008a" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adcc7f1432d4438843d1b2f9d7dd289"
I20260924 12:46:06.761336 4419 raft_consensus.cc:3037] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.761330 4559 raft_consensus.cc:3037] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:06.764308 4559 raft_consensus.cc:2454] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 1.
I20260924 12:46:06.764530 4419 raft_consensus.cc:2454] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 1.
I20260924 12:46:06.764950 4209 leader_election.cc:302] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 6d765d74811f4955a4f0fbaafb2325e8; no voters:
I20260924 12:46:06.765509 4635 raft_consensus.cc:2781] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:06.766057 4635 raft_consensus.cc:686] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Running, Role: LEADER
I20260924 12:46:06.766462 4635 consensus_queue.cc:237] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:06.769730 4120 catalog_manager.cc:5495] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "6d765d74811f4955a4f0fbaafb2325e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:06.801524 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.795707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.2.81.65:42555)
0924 12:46:06.796302 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.796306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.796324 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:06.796404 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.796408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.796483 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.796629 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:06.797089 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.797892 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.799273 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.799696 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.799809 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.800247 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.800845 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.801010 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.801126 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.801207 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":353,"threads_started":1}
I20260924 12:46:06.818809 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.813126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:57223 (local address 127.2.81.66:46765)
0924 12:46:06.813289 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.813293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.814007 (+ 714us) server_negotiation.cc:407] Connection header received
0924 12:46:06.814184 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.814189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.814264 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.814339 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:06.814849 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.815631 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.817041 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.817497 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.817619 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.818057 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.818191 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.818272 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.818406 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.818493 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 12:46:06.818879 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.813122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.2.81.65:57223)
0924 12:46:06.813860 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.813877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.814044 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.814370 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.814374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.814386 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:06.814674 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.814684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.815769 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.815774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.816856 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.816869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.817679 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.817685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.817813 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.817876 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.818285 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.818297 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.818372 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.818531 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":461,"threads_started":1}
I20260924 12:46:06.829017 4419 raft_consensus.cc:1261] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Refusing update from remote peer 6d765d74811f4955a4f0fbaafb2325e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:06.829687 4559 raft_consensus.cc:1261] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Refusing update from remote peer 6d765d74811f4955a4f0fbaafb2325e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:06.829834 4678 consensus_queue.cc:1035] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:06.830276 4635 consensus_queue.cc:1035] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:06.878569 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.871623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42555 (local address 127.2.81.66:35991)
0924 12:46:06.872065 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:06.872085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:06.872209 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:06.873138 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:06.873143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:06.873156 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:06.873405 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.873418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.874689 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.874694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.875968 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:06.875982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.877118 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.877125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:06.877326 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.877379 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.877855 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.877867 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 12:46:06.877932 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 12:46:06.878236 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260924 12:46:06.879307 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:06.872512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:35991 (local address 127.2.81.65:42555)
0924 12:46:06.872677 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:06.872683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:06.872700 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:46:06.872770 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:06.872775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:06.872852 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:06.872933 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:06.873587 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.874551 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.876121 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:06.876821 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:06.876952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:06.877576 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:06.877722 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:06.877809 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:06.878879 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0924 12:46:06.878963 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 12:46:06.880645 4655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 12:46:08.513141 4121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:08.513388 4121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:08.518042 4121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Sending DeleteTablet for 3 replicas of tablet cccd85c001fa4753989610cfa35c008a
I20260924 12:46:08.518965 4259 tablet_service.cc:1514] Processing DeleteTablet for tablet cccd85c001fa4753989610cfa35c008a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:08 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58090
I20260924 12:46:08.518985 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet cccd85c001fa4753989610cfa35c008a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:08 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:36768
I20260924 12:46:08.518985 4399 tablet_service.cc:1514] Processing DeleteTablet for tablet cccd85c001fa4753989610cfa35c008a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:08 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58028
I20260924 12:46:08.519690 4711 tablet_replica.cc:330] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: stopping tablet replica
I20260924 12:46:08.519975 4711 raft_consensus.cc:2229] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:08.520285 4711 raft_consensus.cc:2258] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:08.521484 4711 ts_tablet_manager.cc:1905] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:08.525669 4712 tablet_replica.cc:330] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: stopping tablet replica
I20260924 12:46:08.525827 4710 tablet_replica.cc:330] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: stopping tablet replica
I20260924 12:46:08.525877 4712 raft_consensus.cc:2229] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:08.526051 4710 raft_consensus.cc:2229] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:08.526105 4712 raft_consensus.cc:2258] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:08.526636 4710 raft_consensus.cc:2258] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:08.527650 4712 ts_tablet_manager.cc:1905] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:08.529206 4710 ts_tablet_manager.cc:1905] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 12:46:08 dist-test-slave-9tm1 krb5kdc[4077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260924 12:46:08.559525 4711 ts_tablet_manager.cc:1918] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:46:08.559528 4712 ts_tablet_manager.cc:1918] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:46:08.559716 4711 log.cc:1196] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/cccd85c001fa4753989610cfa35c008a
I20260924 12:46:08.559725 4712 log.cc:1196] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/cccd85c001fa4753989610cfa35c008a
I20260924 12:46:08.560141 4712 ts_tablet_manager.cc:1939] T cccd85c001fa4753989610cfa35c008a P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting consensus metadata
I20260924 12:46:08.560175 4711 ts_tablet_manager.cc:1939] T cccd85c001fa4753989610cfa35c008a P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting consensus metadata
I20260924 12:46:08.560374 4710 ts_tablet_manager.cc:1918] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:46:08.560606 4710 log.cc:1196] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/cccd85c001fa4753989610cfa35c008a
I20260924 12:46:08.561190 4710 ts_tablet_manager.cc:1939] T cccd85c001fa4753989610cfa35c008a P 6d765d74811f4955a4f0fbaafb2325e8: Deleting consensus metadata
I20260924 12:46:08.561614 4106 catalog_manager.cc:4841] TS 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615): tablet cccd85c001fa4753989610cfa35c008a (table test-table [id=e87c5fe3bff143819c08090d47f79880]) successfully deleted
I20260924 12:46:08.561614 4107 catalog_manager.cc:4841] TS 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765): tablet cccd85c001fa4753989610cfa35c008a (table test-table [id=e87c5fe3bff143819c08090d47f79880]) successfully deleted
Sep 24 12:46:08 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:08.575124 4108 catalog_manager.cc:4841] TS 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555): tablet cccd85c001fa4753989610cfa35c008a (table test-table [id=e87c5fe3bff143819c08090d47f79880]) successfully deleted
I20260924 12:46:08.579576 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.563430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.2.81.126:35197)
0924 12:46:08.563921 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:08.563926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:08.563948 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:46:08.564111 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:08.564115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:08.564179 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:08.564269 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:08.565655 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.566497 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.567781 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.568172 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.568287 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.572508 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:08.572514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:08.572518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:08.572560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:08.576215 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:08.577120 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:08.577124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:08.577127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:08.577289 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:08.577761 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:08.577766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:08.577768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:08.578434 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:08.578647 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:08.579024 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 12:46:08.579221 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
I20260924 12:46:08.584163 4121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:08.585143 4121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:08.594303 4399 tablet_service.cc:1467] Processing CreateTablet for tablet ac8e84fa7cf24c93a8e577ef9f224412 (DEFAULT_TABLE table=test-table [id=ad622c570e91467696ae97cbe2419d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:08.594888 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8e84fa7cf24c93a8e577ef9f224412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:08.595260 4537 tablet_service.cc:1467] Processing CreateTablet for tablet ac8e84fa7cf24c93a8e577ef9f224412 (DEFAULT_TABLE table=test-table [id=ad622c570e91467696ae97cbe2419d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:08.595327 4259 tablet_service.cc:1467] Processing CreateTablet for tablet ac8e84fa7cf24c93a8e577ef9f224412 (DEFAULT_TABLE table=test-table [id=ad622c570e91467696ae97cbe2419d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:08.595670 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8e84fa7cf24c93a8e577ef9f224412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:08.595726 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8e84fa7cf24c93a8e577ef9f224412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:08.601065 4721 tablet_bootstrap.cc:492] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Bootstrap starting.
I20260924 12:46:08.602319 4722 tablet_bootstrap.cc:492] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Bootstrap starting.
I20260924 12:46:08.602890 4721 tablet_bootstrap.cc:654] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:08.604489 4722 tablet_bootstrap.cc:654] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:08.606901 4723 tablet_bootstrap.cc:492] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Bootstrap starting.
I20260924 12:46:08.608963 4723 tablet_bootstrap.cc:654] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:08.611415 4722 tablet_bootstrap.cc:492] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: No bootstrap required, opened a new log
I20260924 12:46:08.611450 4723 tablet_bootstrap.cc:492] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: No bootstrap required, opened a new log
I20260924 12:46:08.611598 4723 ts_tablet_manager.cc:1397] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 12:46:08.611599 4722 ts_tablet_manager.cc:1397] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260924 12:46:08.611729 4721 tablet_bootstrap.cc:492] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: No bootstrap required, opened a new log
I20260924 12:46:08.611980 4721 ts_tablet_manager.cc:1397] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260924 12:46:08.612323 4723 raft_consensus.cc:348] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.612421 4722 raft_consensus.cc:348] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.612679 4723 raft_consensus.cc:374] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:08.612749 4722 raft_consensus.cc:374] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:08.612829 4723 raft_consensus.cc:729] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b7a9eaefe4497da5d8a38e5b14ac73, State: Initialized, Role: FOLLOWER
I20260924 12:46:08.612917 4722 raft_consensus.cc:729] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Initialized, Role: FOLLOWER
I20260924 12:46:08.612794 4721 raft_consensus.cc:348] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.613085 4721 raft_consensus.cc:374] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:08.613183 4721 raft_consensus.cc:729] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Initialized, Role: FOLLOWER
I20260924 12:46:08.613190 4723 consensus_queue.cc:260] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.613376 4722 consensus_queue.cc:260] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.613461 4721 consensus_queue.cc:260] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.614117 4723 ts_tablet_manager.cc:1428] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:08.614733 4721 ts_tablet_manager.cc:1428] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:08.615392 4722 ts_tablet_manager.cc:1428] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 12:46:08.820034 4727 raft_consensus.cc:482] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:08.820300 4727 raft_consensus.cc:504] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.821852 4727 leader_election.cc:288] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555), 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765)
I20260924 12:46:08.845829 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.822195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.2.81.67:37947)
0924 12:46:08.838499 (+ 16304us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:08.838523 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:08.838682 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:08.840360 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:08.840365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:08.840404 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:08.840731 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.840744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.842304 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.842310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.843846 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.843860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.844997 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.845003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.845173 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.845210 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 12:46:08.845295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 12:46:08.845446 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16130,"thread_start_us":15906,"threads_started":1}
I20260924 12:46:08.846082 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.822320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:37947 (local address 127.2.81.66:46765)
0924 12:46:08.840041 (+ 17721us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:08.840047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:08.840076 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:46:08.840141 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:08.840145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:08.840205 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:08.840314 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:08.840927 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.842155 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.844021 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.844797 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.844915 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.845092 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:08.845612 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 12:46:08.845756 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17577,"thread_start_us":17365,"threads_started":1}
I20260924 12:46:08.846707 4418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8e84fa7cf24c93a8e577ef9f224412" candidate_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" is_pre_election: true
I20260924 12:46:08.846979 4418 raft_consensus.cc:2454] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18b7a9eaefe4497da5d8a38e5b14ac73 in term 0.
I20260924 12:46:08.847507 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.822164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42555 (local address 127.2.81.67:56365)
0924 12:46:08.838297 (+ 16133us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:08.838326 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:08.838485 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:08.840550 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:08.840555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:08.840573 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:08.841452 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.841466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.842630 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.842636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.844714 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.844729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.846635 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.846641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.846811 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.846844 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:08.846945 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 12:46:08.847081 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15968,"mutex_wait_us":2,"thread_start_us":16387,"threads_started":1}
I20260924 12:46:08.847627 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.822260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:56365 (local address 127.2.81.65:42555)
0924 12:46:08.840243 (+ 17983us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:08.840249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:08.840271 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:46:08.840331 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:08.840336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:08.840403 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:08.840515 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:08.841623 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.842494 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.844904 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.846250 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.846395 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.846512 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:08.847087 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 12:46:08.847300 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17844,"thread_start_us":17696,"threads_started":1}
I20260924 12:46:08.847594 4490 leader_election.cc:302] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:08.848052 4727 raft_consensus.cc:2781] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:08.848170 4727 raft_consensus.cc:482] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:08.848248 4727 raft_consensus.cc:3037] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:08.848330 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8e84fa7cf24c93a8e577ef9f224412" candidate_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d765d74811f4955a4f0fbaafb2325e8" is_pre_election: true
I20260924 12:46:08.848564 4284 raft_consensus.cc:2454] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18b7a9eaefe4497da5d8a38e5b14ac73 in term 0.
I20260924 12:46:08.851613 4727 raft_consensus.cc:504] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.852326 4727 leader_election.cc:288] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [CANDIDATE]: Term 1 election: Requested vote from peers 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555), 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765)
I20260924 12:46:08.852952 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8e84fa7cf24c93a8e577ef9f224412" candidate_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d765d74811f4955a4f0fbaafb2325e8"
I20260924 12:46:08.853058 4418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8e84fa7cf24c93a8e577ef9f224412" candidate_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adcc7f1432d4438843d1b2f9d7dd289"
I20260924 12:46:08.853259 4418 raft_consensus.cc:3037] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:08.853632 4284 raft_consensus.cc:3037] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:08.855427 4418 raft_consensus.cc:2454] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18b7a9eaefe4497da5d8a38e5b14ac73 in term 1.
I20260924 12:46:08.855903 4490 leader_election.cc:302] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:08.856204 4727 raft_consensus.cc:2781] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:08.856660 4727 raft_consensus.cc:686] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 LEADER]: Becoming Leader. State: Replica: 18b7a9eaefe4497da5d8a38e5b14ac73, State: Running, Role: LEADER
I20260924 12:46:08.857061 4284 raft_consensus.cc:2454] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18b7a9eaefe4497da5d8a38e5b14ac73 in term 1.
I20260924 12:46:08.857120 4727 consensus_queue.cc:237] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:08.861222 4119 catalog_manager.cc:5495] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67). New cstate: current_term: 1 leader_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } health_report { overall_health: HEALTHY } } }
I20260924 12:46:08.922601 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.916164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.2.81.67:33615)
0924 12:46:08.917321 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:08.917326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:08.917344 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:08.917430 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:08.917434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:08.917553 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:08.917706 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:08.918144 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.918948 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.920241 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.920669 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.920774 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.921262 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:08.921951 (+ 689us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:08.922095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:08.922166 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 12:46:08.922251 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":882,"threads_started":1}
I20260924 12:46:08.938447 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.932565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:47629 (local address 127.2.81.66:46765)
0924 12:46:08.932722 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:08.932729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:08.933759 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 12:46:08.933964 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:08.933968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:08.934028 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:08.934096 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:08.934599 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.935390 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.936825 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.937269 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.937376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.937798 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:08.937899 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:08.937980 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:08.938100 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 12:46:08.938177 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 12:46:08.938555 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.932481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:46765 (local address 127.2.81.67:47629)
0924 12:46:08.933654 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:08.933671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:08.933820 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:08.934127 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:08.934131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:08.934141 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:08.934436 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.934452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.935512 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.935517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.936698 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.936711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.937436 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.937450 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.937581 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.937634 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:08.937987 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:08.938006 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 12:46:08.938074 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 12:46:08.938205 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":373,"threads_started":1}
I20260924 12:46:08.946822 4418 raft_consensus.cc:1261] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Refusing update from remote peer 18b7a9eaefe4497da5d8a38e5b14ac73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:08.946892 4284 raft_consensus.cc:1261] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Refusing update from remote peer 18b7a9eaefe4497da5d8a38e5b14ac73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:08.947417 4732 consensus_queue.cc:1035] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:08.947780 4727 consensus_queue.cc:1035] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:08.973330 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.967417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:53263 (local address 127.2.81.67:33615)
0924 12:46:08.967572 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:08.967578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:08.967945 (+ 367us) server_negotiation.cc:407] Connection header received
0924 12:46:08.968118 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:08.968123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:08.968198 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:08.968282 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:08.968742 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.969696 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.971562 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.972134 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.972251 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.972643 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:08.972747 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:08.972818 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:08.972932 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 12:46:08.973038 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 12:46:08.973531 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:08.967333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:33615 (local address 127.2.81.66:53263)
0924 12:46:08.967844 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:08.967859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:08.967995 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:08.968301 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:08.968304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:08.968314 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:08.968575 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.968583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.969881 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.969885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.971300 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:08.971382 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:08.972309 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:08.972316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:08.972453 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:08.972490 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:08.972833 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:08.972845 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 12:46:08.972913 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 12:46:08.973089 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":274,"threads_started":1}
I20260924 12:46:09.009248 4119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:09.009608 4119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:09.012653 4119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Sending DeleteTablet for 3 replicas of tablet ac8e84fa7cf24c93a8e577ef9f224412
I20260924 12:46:09.013641 4259 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8e84fa7cf24c93a8e577ef9f224412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:09 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58090
I20260924 12:46:09.013932 4710 tablet_replica.cc:330] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: stopping tablet replica
I20260924 12:46:09.014070 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8e84fa7cf24c93a8e577ef9f224412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:09 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:36768
I20260924 12:46:09.014119 4710 raft_consensus.cc:2229] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:09.014307 4710 raft_consensus.cc:2258] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:09.014333 4712 tablet_replica.cc:330] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: stopping tablet replica
I20260924 12:46:09.014585 4712 raft_consensus.cc:2229] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:09.014776 4119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 12:46:09.015080 4712 raft_consensus.cc:2258] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:09.015694 4710 ts_tablet_manager.cc:1905] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 12:46:09.015736 4119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:09.016330 4712 ts_tablet_manager.cc:1905] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:09.021502 4399 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8e84fa7cf24c93a8e577ef9f224412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:09 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58028
I20260924 12:46:09.021919 4711 tablet_replica.cc:330] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: stopping tablet replica
I20260924 12:46:09.022146 4711 raft_consensus.cc:2229] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:09.022339 4711 raft_consensus.cc:2258] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:09.023676 4711 ts_tablet_manager.cc:1905] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:09.023896 4537 tablet_service.cc:1467] Processing CreateTablet for tablet 049ec1ae6b374805bcebf75f705e95d4 (DEFAULT_TABLE table=test-table [id=92872ca1e18f4dc4afa1ccce63d51fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:09.024341 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049ec1ae6b374805bcebf75f705e95d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:09.029070 4712 ts_tablet_manager.cc:1918] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:09.029238 4712 log.cc:1196] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/ac8e84fa7cf24c93a8e577ef9f224412
I20260924 12:46:09.029695 4712 ts_tablet_manager.cc:1939] T ac8e84fa7cf24c93a8e577ef9f224412 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting consensus metadata
I20260924 12:46:09.029677 4259 tablet_service.cc:1467] Processing CreateTablet for tablet 049ec1ae6b374805bcebf75f705e95d4 (DEFAULT_TABLE table=test-table [id=92872ca1e18f4dc4afa1ccce63d51fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:09.030098 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049ec1ae6b374805bcebf75f705e95d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:09.030787 4106 catalog_manager.cc:4841] TS 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615): tablet ac8e84fa7cf24c93a8e577ef9f224412 (table test-table [id=ad622c570e91467696ae97cbe2419d2b]) successfully deleted
I20260924 12:46:09.031606 4723 tablet_bootstrap.cc:492] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Bootstrap starting.
I20260924 12:46:09.033080 4723 tablet_bootstrap.cc:654] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:09.034884 4399 tablet_service.cc:1467] Processing CreateTablet for tablet 049ec1ae6b374805bcebf75f705e95d4 (DEFAULT_TABLE table=test-table [id=92872ca1e18f4dc4afa1ccce63d51fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:09.035312 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049ec1ae6b374805bcebf75f705e95d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:09.036643 4721 tablet_bootstrap.cc:492] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Bootstrap starting.
I20260924 12:46:09.038208 4721 tablet_bootstrap.cc:654] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:09.038651 4711 ts_tablet_manager.cc:1918] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:09.038794 4711 log.cc:1196] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/ac8e84fa7cf24c93a8e577ef9f224412
I20260924 12:46:09.039247 4711 ts_tablet_manager.cc:1939] T ac8e84fa7cf24c93a8e577ef9f224412 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting consensus metadata
I20260924 12:46:09.040297 4107 catalog_manager.cc:4841] TS 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765): tablet ac8e84fa7cf24c93a8e577ef9f224412 (table test-table [id=ad622c570e91467696ae97cbe2419d2b]) successfully deleted
I20260924 12:46:09.041297 4710 ts_tablet_manager.cc:1918] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:09.041484 4710 log.cc:1196] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/ac8e84fa7cf24c93a8e577ef9f224412
I20260924 12:46:09.042217 4710 ts_tablet_manager.cc:1939] T ac8e84fa7cf24c93a8e577ef9f224412 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting consensus metadata
I20260924 12:46:09.042388 4722 tablet_bootstrap.cc:492] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Bootstrap starting.
I20260924 12:46:09.043330 4108 catalog_manager.cc:4841] TS 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555): tablet ac8e84fa7cf24c93a8e577ef9f224412 (table test-table [id=ad622c570e91467696ae97cbe2419d2b]) successfully deleted
I20260924 12:46:09.044765 4723 tablet_bootstrap.cc:492] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: No bootstrap required, opened a new log
I20260924 12:46:09.044795 4722 tablet_bootstrap.cc:654] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:09.044947 4723 ts_tablet_manager.cc:1397] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20260924 12:46:09.045917 4723 raft_consensus.cc:348] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.046200 4723 raft_consensus.cc:374] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:09.046288 4723 raft_consensus.cc:729] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b7a9eaefe4497da5d8a38e5b14ac73, State: Initialized, Role: FOLLOWER
I20260924 12:46:09.046572 4723 consensus_queue.cc:260] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.048136 4723 ts_tablet_manager.cc:1428] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 12:46:09.053926 4721 tablet_bootstrap.cc:492] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: No bootstrap required, opened a new log
I20260924 12:46:09.054023 4722 tablet_bootstrap.cc:492] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: No bootstrap required, opened a new log
I20260924 12:46:09.054118 4721 ts_tablet_manager.cc:1397] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.002s
I20260924 12:46:09.054196 4722 ts_tablet_manager.cc:1397] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260924 12:46:09.054998 4722 raft_consensus.cc:348] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.055264 4722 raft_consensus.cc:374] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:09.055332 4722 raft_consensus.cc:729] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adcc7f1432d4438843d1b2f9d7dd289, State: Initialized, Role: FOLLOWER
I20260924 12:46:09.055291 4721 raft_consensus.cc:348] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.055569 4721 raft_consensus.cc:374] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:09.055647 4721 raft_consensus.cc:729] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Initialized, Role: FOLLOWER
I20260924 12:46:09.055588 4722 consensus_queue.cc:260] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.056183 4722 ts_tablet_manager.cc:1428] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:46:09.056506 4721 consensus_queue.cc:260] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.057220 4721 ts_tablet_manager.cc:1428] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 12:46:09.309603 4686 raft_consensus.cc:482] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:09.309816 4686 raft_consensus.cc:504] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.310521 4686 leader_election.cc:288] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765), 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:09.311107 4559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049ec1ae6b374805bcebf75f705e95d4" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" is_pre_election: true
I20260924 12:46:09.311209 4418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049ec1ae6b374805bcebf75f705e95d4" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" is_pre_election: true
I20260924 12:46:09.311414 4559 raft_consensus.cc:2454] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 0.
I20260924 12:46:09.311509 4418 raft_consensus.cc:2454] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 0.
I20260924 12:46:09.311985 4210 leader_election.cc:302] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d765d74811f4955a4f0fbaafb2325e8, 8adcc7f1432d4438843d1b2f9d7dd289; no voters:
I20260924 12:46:09.312264 4686 raft_consensus.cc:2781] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:09.312403 4686 raft_consensus.cc:482] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:09.312496 4686 raft_consensus.cc:3037] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:09.314741 4686 raft_consensus.cc:504] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.315351 4686 leader_election.cc:288] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 election: Requested vote from peers 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765), 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615)
I20260924 12:46:09.315984 4559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049ec1ae6b374805bcebf75f705e95d4" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73"
I20260924 12:46:09.315986 4418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049ec1ae6b374805bcebf75f705e95d4" candidate_uuid: "6d765d74811f4955a4f0fbaafb2325e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8adcc7f1432d4438843d1b2f9d7dd289"
I20260924 12:46:09.316210 4559 raft_consensus.cc:3037] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:09.316223 4418 raft_consensus.cc:3037] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:09.319988 4559 raft_consensus.cc:2454] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 1.
I20260924 12:46:09.319989 4418 raft_consensus.cc:2454] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d765d74811f4955a4f0fbaafb2325e8 in term 1.
I20260924 12:46:09.320571 4209 leader_election.cc:302] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b7a9eaefe4497da5d8a38e5b14ac73, 6d765d74811f4955a4f0fbaafb2325e8; no voters:
I20260924 12:46:09.320919 4686 raft_consensus.cc:2781] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:09.321569 4686 raft_consensus.cc:686] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d765d74811f4955a4f0fbaafb2325e8, State: Running, Role: LEADER
I20260924 12:46:09.321983 4686 consensus_queue.cc:237] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } }
I20260924 12:46:09.325134 4119 catalog_manager.cc:5495] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "6d765d74811f4955a4f0fbaafb2325e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d765d74811f4955a4f0fbaafb2325e8" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:09.355964 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:09.349615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.2.81.65:42555)
0924 12:46:09.350125 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:09.350132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:09.350149 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:46:09.350204 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:09.350207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:09.350270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:09.350367 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:09.350759 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:09.351714 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:09.353214 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:09.353613 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:09.354138 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:09.354936 (+ 798us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:09.355068 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:09.355148 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:09.355478 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 12:46:09.355557 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":290,"threads_started":1}
I20260924 12:46:09.368377 4559 raft_consensus.cc:1261] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Refusing update from remote peer 6d765d74811f4955a4f0fbaafb2325e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:09.368399 4418 raft_consensus.cc:1261] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Refusing update from remote peer 6d765d74811f4955a4f0fbaafb2325e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:09.369153 4749 consensus_queue.cc:1035] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8adcc7f1432d4438843d1b2f9d7dd289" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:09.369602 4686 consensus_queue.cc:1035] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18b7a9eaefe4497da5d8a38e5b14ac73" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:46:09.407546 4694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 12:46:11.008232 4119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:11.008476 4119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:11.012668 4119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8a88eede424f6cb346c6d8b0eb97b8: Sending DeleteTablet for 3 replicas of tablet 049ec1ae6b374805bcebf75f705e95d4
I20260924 12:46:11.013585 4259 tablet_service.cc:1514] Processing DeleteTablet for tablet 049ec1ae6b374805bcebf75f705e95d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:11 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58090
I20260924 12:46:11.013619 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet 049ec1ae6b374805bcebf75f705e95d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:11 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:36768
I20260924 12:46:11.013635 4399 tablet_service.cc:1514] Processing DeleteTablet for tablet 049ec1ae6b374805bcebf75f705e95d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:11 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:58028
I20260924 12:46:11.014340 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4193
I20260924 12:46:11.029960 4779 tablet_replica.cc:330] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: stopping tablet replica
I20260924 12:46:11.030120 4780 tablet_replica.cc:330] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: stopping tablet replica
I20260924 12:46:11.030339 4780 raft_consensus.cc:2229] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:11.030393 4779 raft_consensus.cc:2229] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:11.030650 4780 raft_consensus.cc:2258] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:11.030649 4779 raft_consensus.cc:2258] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:11.032353 4780 ts_tablet_manager.cc:1905] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:11.032693 4778 tablet_replica.cc:330] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: stopping tablet replica
I20260924 12:46:11.032724 4779 ts_tablet_manager.cc:1905] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:11.032902 4778 raft_consensus.cc:2229] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:11.033281 4778 raft_consensus.cc:2258] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:11.034281 4778 ts_tablet_manager.cc:1905] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:11.039619 4780 ts_tablet_manager.cc:1918] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:46:11.039783 4780 log.cc:1196] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/049ec1ae6b374805bcebf75f705e95d4
I20260924 12:46:11.039826 4779 ts_tablet_manager.cc:1918] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:46:11.039996 4779 log.cc:1196] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/049ec1ae6b374805bcebf75f705e95d4
I20260924 12:46:11.040254 4780 ts_tablet_manager.cc:1939] T 049ec1ae6b374805bcebf75f705e95d4 P 18b7a9eaefe4497da5d8a38e5b14ac73: Deleting consensus metadata
I20260924 12:46:11.040417 4779 ts_tablet_manager.cc:1939] T 049ec1ae6b374805bcebf75f705e95d4 P 8adcc7f1432d4438843d1b2f9d7dd289: Deleting consensus metadata
I20260924 12:46:11.041131 4778 ts_tablet_manager.cc:1918] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 12:46:11.041325 4778 log.cc:1196] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/049ec1ae6b374805bcebf75f705e95d4
I20260924 12:46:11.041373 4107 catalog_manager.cc:4841] TS 8adcc7f1432d4438843d1b2f9d7dd289 (127.2.81.66:46765): tablet 049ec1ae6b374805bcebf75f705e95d4 (table test-table [id=92872ca1e18f4dc4afa1ccce63d51fc1]) successfully deleted
I20260924 12:46:11.041639 4106 catalog_manager.cc:4841] TS 18b7a9eaefe4497da5d8a38e5b14ac73 (127.2.81.67:33615): tablet 049ec1ae6b374805bcebf75f705e95d4 (table test-table [id=92872ca1e18f4dc4afa1ccce63d51fc1]) successfully deleted
I20260924 12:46:11.041736 4778 ts_tablet_manager.cc:1939] T 049ec1ae6b374805bcebf75f705e95d4 P 6d765d74811f4955a4f0fbaafb2325e8: Deleting consensus metadata
I20260924 12:46:11.042615 4108 catalog_manager.cc:4841] TS 6d765d74811f4955a4f0fbaafb2325e8 (127.2.81.65:42555): tablet 049ec1ae6b374805bcebf75f705e95d4 (table test-table [id=92872ca1e18f4dc4afa1ccce63d51fc1]) successfully deleted
Sep 24 12:46:11 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:11.052942 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:11.015412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.2.81.65:42555)
0924 12:46:11.031869 (+ 16457us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:11.031876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:11.031907 (+ 31us) server_negotiation.cc:407] Connection header received
0924 12:46:11.031978 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:11.031984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:11.032087 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:11.032216 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:11.034091 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:11.035233 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:11.037019 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:11.037668 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:11.037793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:11.046016 (+ 8223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:11.046036 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:11.046057 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:11.046103 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:11.049360 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:11.050115 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:11.050123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:11.050125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:11.050270 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:11.050661 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:11.050665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:11.050667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:11.051215 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:11.051773 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:11.052145 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 12:46:11.052589 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16305,"thread_start_us":16169,"threads_started":1}
W20260924 12:46:11.053829 4324 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58122
W20260924 12:46:11.054525 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4193: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:11.054808 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 4193
I20260924 12:46:11.073122 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4333
Sep 24 12:46:11 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:46:11.088035 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:11.073797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.2.81.66:46765)
0924 12:46:11.074247 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:11.074252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:11.074268 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:46:11.074320 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:11.074324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:11.074385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:11.074516 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:11.075708 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:11.076444 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:11.077670 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:11.078147 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:11.078257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:11.082294 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:11.082312 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:11.082329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:11.082369 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:11.085371 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:11.086107 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:11.086115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:11.086117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:11.086287 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:11.086630 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:11.086634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:11.086637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:11.087043 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:11.087250 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:11.087521 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 12:46:11.087689 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
W20260924 12:46:11.088675 4464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074
W20260924 12:46:11.089339 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:11.089498 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 4333
I20260924 12:46:11.110292 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4473
Sep 24 12:46:11 dist-test-slave-9tm1 krb5kdc[4077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:46:11.125873 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:11.110970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.2.81.67:33615)
0924 12:46:11.111448 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:11.111454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:11.111472 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:11.111526 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:11.111530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:11.111591 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:11.111693 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:11.112894 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:11.113654 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:11.115068 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:11.115497 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:11.115607 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:11.120089 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:11.120104 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:11.120119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:11.120156 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:11.122952 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:11.123807 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:11.123813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:11.123816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:11.124002 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:11.124356 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:11.124358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:11.124360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:11.124702 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:11.124917 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:11.125410 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 12:46:11.125581 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
W20260924 12:46:11.126474 4603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36778
W20260924 12:46:11.127069 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:11.127228 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 4473
I20260924 12:46:11.143787 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4090
I20260924 12:46:11.144958 4162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:11.256821 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 4090
2026-09-24T12:46:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8799 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:11 dist-test-slave-9tm1 krb5kdc[4788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:11 dist-test-slave-9tm1 krb5kdc[4788](info): set up 2 sockets
Sep 24 12:46:11 dist-test-slave-9tm1 krb5kdc[4788](info): commencing operation
krb5kdc: starting...
W20260924 12:46:12.273689 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:12 dist-test-slave-9tm1 krb5kdc[4788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:12.467422 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:35769
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:12.799523 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:12.799839 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:12.799940 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:12.807606 4801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:12.807749 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:12.807796 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:12.807847 4801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:12.807893 4801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:12.818295 4801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:12.819617 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:12.820917 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:12.833856 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:12.833768 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:12.833911 4801 server_base.cc:1034] running on GCE node
W20260924 12:46:12.833827 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:12.835074 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:12.836663 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:12.837903 4801 hybrid_clock.cc:648] HybridClock initialized: now 1790253972837842 us; error 52 us; skew 500 ppm
I20260924 12:46:12.838274 4801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:12 dist-test-slave-9tm1 krb5kdc[4788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253972, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:12.841696 4801 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:12.843309 4801 webserver.cc:460] Webserver started at http://127.2.81.126:34909/ using document root <none> and password file <none>
I20260924 12:46:12.844034 4801 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:12.844156 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:12.844461 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:12.847711 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "df8e6b61747d46b6b3769c2da08b768d"
format_stamp: "Formatted at 2026-09-24 12:46:12 on dist-test-slave-9tm1"
server_key: "c7ebc5d660442e9aa1a9a0c8ecc22606"
server_key_iv: "c55504ed09095bbe610092a1db02386f"
server_key_version: "encryptionkey@0"
I20260924 12:46:12.848590 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "df8e6b61747d46b6b3769c2da08b768d"
format_stamp: "Formatted at 2026-09-24 12:46:12 on dist-test-slave-9tm1"
server_key: "c7ebc5d660442e9aa1a9a0c8ecc22606"
server_key_iv: "c55504ed09095bbe610092a1db02386f"
server_key_version: "encryptionkey@0"
I20260924 12:46:12.854584 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260924 12:46:12.858659 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:12.860324 4801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 12:46:12.860530 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "df8e6b61747d46b6b3769c2da08b768d"
format_stamp: "Formatted at 2026-09-24 12:46:12 on dist-test-slave-9tm1"
server_key: "c7ebc5d660442e9aa1a9a0c8ecc22606"
server_key_iv: "c55504ed09095bbe610092a1db02386f"
server_key_version: "encryptionkey@0"
I20260924 12:46:12.860736 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:12.882222 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:12.889272 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:12.889573 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:12.907526 4801 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:35769
I20260924 12:46:12.907554 4867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:35769 every 8 connection(s)
I20260924 12:46:12.908921 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:12.913105 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:12.916489 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 4801
I20260924 12:46:12.916724 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:12.917472 2373 external_mini_cluster.cc:1428] Setting key edc1effc4a6e04b08b838ae2c6e80c2c
I20260924 12:46:12.926704 4868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d: Bootstrap starting.
I20260924 12:46:12.930593 4868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:12.931977 4868 log.cc:824] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:46:12 dist-test-slave-9tm1 krb5kdc[4788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:12.935417 4868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d: No bootstrap required, opened a new log
I20260924 12:46:12.942060 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:12.919830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.2.81.126:35769)
0924 12:46:12.920651 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:12.920663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:12.920707 (+ 44us) server_negotiation.cc:407] Connection header received
0924 12:46:12.922438 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:12.922464 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:12.922939 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:12.923302 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:12.925087 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:12.926584 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:12.928355 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:12.929197 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:12.929342 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:12.934446 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:12.934457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:12.934475 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:12.934515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:12.937712 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:12.938473 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:12.938480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:12.938491 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:12.938672 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:12.939063 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:12.939067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:12.939070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:12.939649 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:12.939958 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:12.940437 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 12:46:12.940898 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":373,"threads_started":1}
I20260924 12:46:12.943392 4868 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8e6b61747d46b6b3769c2da08b768d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35769 } }
I20260924 12:46:12.943766 4868 raft_consensus.cc:374] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:12.943859 4868 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e6b61747d46b6b3769c2da08b768d, State: Initialized, Role: FOLLOWER
I20260924 12:46:12.944442 4868 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8e6b61747d46b6b3769c2da08b768d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35769 } }
I20260924 12:46:12.944676 4868 raft_consensus.cc:388] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:12.944815 4868 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:12.945031 4868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:12.947672 4868 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8e6b61747d46b6b3769c2da08b768d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35769 } }
I20260924 12:46:12.948215 4868 leader_election.cc:302] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df8e6b61747d46b6b3769c2da08b768d; no voters:
I20260924 12:46:12.948906 4868 leader_election.cc:288] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:12.949121 4873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:12.949858 4873 raft_consensus.cc:686] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [term 1 LEADER]: Becoming Leader. State: Replica: df8e6b61747d46b6b3769c2da08b768d, State: Running, Role: LEADER
I20260924 12:46:12.950644 4873 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8e6b61747d46b6b3769c2da08b768d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35769 } }
I20260924 12:46:12.951821 4868 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:12.953477 4874 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8e6b61747d46b6b3769c2da08b768d. Latest consensus state: current_term: 1 leader_uuid: "df8e6b61747d46b6b3769c2da08b768d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8e6b61747d46b6b3769c2da08b768d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35769 } } }
I20260924 12:46:12.953897 4874 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:12.953452 4875 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df8e6b61747d46b6b3769c2da08b768d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df8e6b61747d46b6b3769c2da08b768d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35769 } } }
I20260924 12:46:12.954591 4875 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:12.957424 4879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:12.962610 4879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:12.973815 4879 catalog_manager.cc:1347] Generated new cluster ID: d58e1a3d1fe24cf4ac0eea4bbf2d0e11
I20260924 12:46:12.973966 4879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:12.996634 4879 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:12.998108 4879 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:13.006151 4879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df8e6b61747d46b6b3769c2da08b768d: Generated new TSK 0
I20260924 12:46:13.007390 4879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:46:13.068776 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35769
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:13.432611 4894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:13.433109 4894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:13.433279 4894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:13.440964 4894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:13.441145 4894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:13.441339 4894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:13.453161 4894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:13.454465 4894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:13.455991 4894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:13.470537 4902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:13.470641 4899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:13.470446 4900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:13.476917 4894 server_base.cc:1034] running on GCE node
I20260924 12:46:13.477717 4894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:13.478822 4894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:13.480072 4894 hybrid_clock.cc:648] HybridClock initialized: now 1790253973479997 us; error 70 us; skew 500 ppm
I20260924 12:46:13.480551 4894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:13 dist-test-slave-9tm1 krb5kdc[4788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253973, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:13.483863 4894 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:46:13.485574 4894 webserver.cc:460] Webserver started at http://127.2.81.65:37351/ using document root <none> and password file <none>
I20260924 12:46:13.486297 4894 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:13.486410 4894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:13.486683 4894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:13.489801 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "ca5e8e3be81d4d4ab547925601b94296"
format_stamp: "Formatted at 2026-09-24 12:46:13 on dist-test-slave-9tm1"
server_key: "86390911e31a36c88e07f64fa0802639"
server_key_iv: "7a2aa01c8be4841fba8787177c28fba7"
server_key_version: "encryptionkey@0"
I20260924 12:46:13.490610 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "ca5e8e3be81d4d4ab547925601b94296"
format_stamp: "Formatted at 2026-09-24 12:46:13 on dist-test-slave-9tm1"
server_key: "86390911e31a36c88e07f64fa0802639"
server_key_iv: "7a2aa01c8be4841fba8787177c28fba7"
server_key_version: "encryptionkey@0"
I20260924 12:46:13.496345 4894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:46:13.501008 4909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:13.502902 4894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:46:13.503118 4894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "ca5e8e3be81d4d4ab547925601b94296"
format_stamp: "Formatted at 2026-09-24 12:46:13 on dist-test-slave-9tm1"
server_key: "86390911e31a36c88e07f64fa0802639"
server_key_iv: "7a2aa01c8be4841fba8787177c28fba7"
server_key_version: "encryptionkey@0"
I20260924 12:46:13.503334 4894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:13.517184 4894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:13.521538 4894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:13.521801 4894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:13.522740 4894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:13.524081 4894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:13.524153 4894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:13.524240 4894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:13.524283 4894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:13.563999 4894 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:44947
I20260924 12:46:13.564088 5021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:44947 every 8 connection(s)
I20260924 12:46:13.565510 4894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:13.570845 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 4894
I20260924 12:46:13.571076 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:13.572423 2373 external_mini_cluster.cc:1428] Setting key ac13233bc9301ce2a42ddc658aaa0c13
Sep 24 12:46:13 dist-test-slave-9tm1 krb5kdc[4788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253973, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:46:13.593792 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:13.568322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:60319 (local address 127.2.81.126:35769)
0924 12:46:13.568875 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:13.568880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:13.570128 (+ 1248us) server_negotiation.cc:407] Connection header received
0924 12:46:13.572666 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:13.572674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:13.573057 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:13.573206 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:13.576235 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:13.577838 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:13.580391 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:13.581074 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:13.581238 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:13.586324 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:13.586330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:13.586334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:13.586376 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:13.589659 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:13.590927 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:13.590935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:13.590938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:13.591122 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:13.591731 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:13.591736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:13.591741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:13.592134 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:13.592287 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:13.593347 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 12:46:13.593538 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":344,"threads_started":1}
I20260924 12:46:13.594914 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:13.568832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35769 (local address 127.2.81.65:60319)
0924 12:46:13.569903 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:13.569958 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:13.572265 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:13.573648 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:13.573664 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:13.574464 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:13.575586 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:13.575627 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:13.578220 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:13.578227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:13.579978 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:13.580016 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:13.581499 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:13.581522 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:13.581877 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:13.582748 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:13.582784 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:13.586008 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:13.589889 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:13.589898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:13.589917 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:13.590708 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:13.591281 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:13.591285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:13.591287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:13.591596 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:13.592352 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:13.592362 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:13.592868 (+ 506us) client_negotiation.cc:769] Sending connection context
0924 12:46:13.593233 (+ 365us) client_negotiation.cc:240] Negotiation successful
0924 12:46:13.593679 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":531,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:46:13.596752 5022 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35769
I20260924 12:46:13.597111 5022 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:13.597893 5022 heartbeater.cc:511] Master 127.2.81.126:35769 requested a full tablet report, sending...
I20260924 12:46:13.599980 4833 ts_manager.cc:194] Registered new tserver with Master: ca5e8e3be81d4d4ab547925601b94296 (127.2.81.65:44947)
I20260924 12:46:13.601794 4833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:60319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:13.621165 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35769
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:13.963606 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:13.963925 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:13.964102 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:13.971676 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:13.971812 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:13.971982 5028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:13.982048 5028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:13.983376 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:13.984712 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:13.997999 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:13.998029 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:13.998167 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:13.999171 5028 server_base.cc:1034] running on GCE node
I20260924 12:46:13.999799 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:14.000782 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:14.002012 5028 hybrid_clock.cc:648] HybridClock initialized: now 1790253974001949 us; error 55 us; skew 500 ppm
I20260924 12:46:14.002337 5028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:13 dist-test-slave-9tm1 krb5kdc[4788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253973, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:14.005468 5028 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:14.007131 5028 webserver.cc:460] Webserver started at http://127.2.81.66:36539/ using document root <none> and password file <none>
I20260924 12:46:14.007843 5028 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:14.007962 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:14.008281 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:14.011294 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "33a1a07f62324b6d9bfdc16794bd5fa3"
format_stamp: "Formatted at 2026-09-24 12:46:14 on dist-test-slave-9tm1"
server_key: "1bc831d20781ea3f6dbdfc6caf698757"
server_key_iv: "7f210bca217407609d0b384016b06745"
server_key_version: "encryptionkey@0"
I20260924 12:46:14.012126 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "33a1a07f62324b6d9bfdc16794bd5fa3"
format_stamp: "Formatted at 2026-09-24 12:46:14 on dist-test-slave-9tm1"
server_key: "1bc831d20781ea3f6dbdfc6caf698757"
server_key_iv: "7f210bca217407609d0b384016b06745"
server_key_version: "encryptionkey@0"
I20260924 12:46:14.017653 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 12:46:14.021394 5043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:14.022940 5028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:14.023113 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "33a1a07f62324b6d9bfdc16794bd5fa3"
format_stamp: "Formatted at 2026-09-24 12:46:14 on dist-test-slave-9tm1"
server_key: "1bc831d20781ea3f6dbdfc6caf698757"
server_key_iv: "7f210bca217407609d0b384016b06745"
server_key_version: "encryptionkey@0"
I20260924 12:46:14.023313 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:14.047441 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:14.052506 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:14.052811 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:14.053851 5028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:14.055225 5028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:14.055305 5028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:14.055424 5028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:14.055480 5028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:14.087195 5028 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:41395
I20260924 12:46:14.087265 5155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:41395 every 8 connection(s)
I20260924 12:46:14.088506 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:14.089781 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5028
I20260924 12:46:14.089970 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:14.090606 2373 external_mini_cluster.cc:1428] Setting key 31e21bf82dabc0154797d6468543ad7d
Sep 24 12:46:14 dist-test-slave-9tm1 krb5kdc[4788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253973, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:46:14.112066 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:14.091502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:40479 (local address 127.2.81.126:35769)
0924 12:46:14.091718 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:14.091724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:14.093188 (+ 1464us) server_negotiation.cc:407] Connection header received
0924 12:46:14.095695 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:14.095702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:14.095782 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:14.095873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:14.098020 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.098899 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.100773 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.101471 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.101608 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:14.105158 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:14.105165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:14.105172 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:14.105216 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:14.108462 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:14.109451 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:14.109457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:14.109459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:14.109662 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:14.110139 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:14.110143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:14.110145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:14.110442 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:14.110577 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:14.111407 (+ 830us) server_negotiation.cc:299] Negotiation successful
0924 12:46:14.111656 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:46:14.113330 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:14.092119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35769 (local address 127.2.81.66:40479)
0924 12:46:14.093033 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:14.093083 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:14.095328 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:14.096160 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:14.096173 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:14.096925 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:14.097785 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:14.097810 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.099078 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.099083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:14.100548 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:14.100563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.101696 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.101701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:14.101921 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:14.102574 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:14.102598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:14.104839 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:14.108678 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:14.108688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:14.108706 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:14.109291 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:14.109793 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:14.109797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:14.109799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:14.110012 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:14.110597 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:14.110607 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:14.110990 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 12:46:14.111279 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 12:46:14.111891 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":385,"threads_started":1}
I20260924 12:46:14.114984 5156 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35769
I20260924 12:46:14.115305 5156 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:14.116073 5156 heartbeater.cc:511] Master 127.2.81.126:35769 requested a full tablet report, sending...
I20260924 12:46:14.117797 4833 ts_manager.cc:194] Registered new tserver with Master: 33a1a07f62324b6d9bfdc16794bd5fa3 (127.2.81.66:41395)
I20260924 12:46:14.118897 4833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:40479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:14.140509 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35769
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:14.465183 5161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:14.465526 5161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:14.465670 5161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:14.473539 5161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:14.473675 5161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:14.473819 5161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:14.484040 5161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:14.485479 5161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:14.486857 5161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:14.500746 5167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:14.500717 5169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:14.500831 5166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:14.501381 5161 server_base.cc:1034] running on GCE node
I20260924 12:46:14.502048 5161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:14.503077 5161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:14.504292 5161 hybrid_clock.cc:648] HybridClock initialized: now 1790253974504228 us; error 65 us; skew 500 ppm
I20260924 12:46:14.504627 5161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:14 dist-test-slave-9tm1 krb5kdc[4788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:14.508322 5161 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:14.510273 5161 webserver.cc:460] Webserver started at http://127.2.81.67:44709/ using document root <none> and password file <none>
I20260924 12:46:14.511098 5161 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:14.511226 5161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:14.511544 5161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:14.514671 5161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "9461f81e8f8c4e5daba09134ee04f7d4"
format_stamp: "Formatted at 2026-09-24 12:46:14 on dist-test-slave-9tm1"
server_key: "4ba7e6e69c417fcb66c9370d3ac44ca1"
server_key_iv: "aa6856b946ca41190a2c62cef920d2cf"
server_key_version: "encryptionkey@0"
I20260924 12:46:14.515535 5161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "9461f81e8f8c4e5daba09134ee04f7d4"
format_stamp: "Formatted at 2026-09-24 12:46:14 on dist-test-slave-9tm1"
server_key: "4ba7e6e69c417fcb66c9370d3ac44ca1"
server_key_iv: "aa6856b946ca41190a2c62cef920d2cf"
server_key_version: "encryptionkey@0"
I20260924 12:46:14.521297 5161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:46:14.525341 5176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:14.526958 5161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:14.527151 5161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "9461f81e8f8c4e5daba09134ee04f7d4"
format_stamp: "Formatted at 2026-09-24 12:46:14 on dist-test-slave-9tm1"
server_key: "4ba7e6e69c417fcb66c9370d3ac44ca1"
server_key_iv: "aa6856b946ca41190a2c62cef920d2cf"
server_key_version: "encryptionkey@0"
I20260924 12:46:14.527357 5161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:14.544371 5161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:14.549069 5161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:14.549350 5161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:14.550343 5161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:14.551710 5161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:14.551788 5161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:14.551906 5161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:14.551971 5161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:14.587451 5161 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:46335
I20260924 12:46:14.587500 5288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:46335 every 8 connection(s)
I20260924 12:46:14.588831 5161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:14.598537 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5161
I20260924 12:46:14.598737 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:14.599177 2373 external_mini_cluster.cc:1428] Setting key 618dccccb66b55e14ce31d2710ee668b
Sep 24 12:46:14 dist-test-slave-9tm1 krb5kdc[4788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253974, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:14.605058 5022 heartbeater.cc:503] Master 127.2.81.126:35769 was elected leader, sending a full tablet report...
I20260924 12:46:14.611142 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:14.593224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:37949 (local address 127.2.81.126:35769)
0924 12:46:14.593428 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:14.593434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:14.593462 (+ 28us) server_negotiation.cc:407] Connection header received
0924 12:46:14.595760 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:14.595766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:14.595894 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:14.595982 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:14.597836 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.598798 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.600690 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.601340 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.601471 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:14.605288 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:14.605294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:14.605298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:14.605334 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:14.608051 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:14.608840 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:14.608845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:14.608847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:14.609025 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:14.609439 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:14.609444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:14.609446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:14.609737 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:14.609857 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:14.610689 (+ 832us) server_negotiation.cc:299] Negotiation successful
0924 12:46:14.610877 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:46:14.612138 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:14.592414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35769 (local address 127.2.81.67:37949)
0924 12:46:14.593311 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:14.593362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:14.595401 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:14.596262 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:14.596272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:14.596898 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:14.597634 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:14.597655 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.598960 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.598964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:14.600550 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:14.600565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:14.601568 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:14.601577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:14.601838 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:14.602618 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:14.602650 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:14.604966 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:14.608218 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:14.608226 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:14.608241 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:14.608724 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:14.609105 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:14.609108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:14.609110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:14.609299 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:14.609877 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:14.609883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:14.610256 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 12:46:14.610508 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 12:46:14.610977 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":324,"threads_started":1}
I20260924 12:46:14.613792 5289 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35769
I20260924 12:46:14.614113 5289 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:14.614872 5289 heartbeater.cc:511] Master 127.2.81.126:35769 requested a full tablet report, sending...
I20260924 12:46:14.616355 4833 ts_manager.cc:194] Registered new tserver with Master: 9461f81e8f8c4e5daba09134ee04f7d4 (127.2.81.67:46335)
I20260924 12:46:14.617249 4833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:37949
I20260924 12:46:14.627771 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 12:46:14.656824 5024 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:14.654325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.2.81.126:35769)
0924 12:46:14.654547 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:14.654554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:14.654896 (+ 342us) server_negotiation.cc:407] Connection header received
0924 12:46:14.655114 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:14.655120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:14.655190 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:14.655271 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:14.656504 (+ 1233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33038: BlockingRecv error: recv got EOF from 127.0.0.1:33038 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 12:46:14.660434 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4894
W20260924 12:46:14.663480 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4894: Not authorized: Client connection negotiation failed: client connection to 127.2.81.65:44947: server requires authentication, but client does not have Kerberos credentials available
I20260924 12:46:14.663697 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 4894
I20260924 12:46:14.675844 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5028
W20260924 12:46:14.678380 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5028: Not authorized: Client connection negotiation failed: client connection to 127.2.81.66:41395: server requires authentication, but client does not have Kerberos credentials available
I20260924 12:46:14.678524 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5028
I20260924 12:46:14.690045 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5161
W20260924 12:46:14.692768 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5161: Not authorized: Client connection negotiation failed: client connection to 127.2.81.67:46335: server requires authentication, but client does not have Kerberos credentials available
I20260924 12:46:14.692905 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5161
I20260924 12:46:14.705036 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 4801
I20260924 12:46:14.706578 4863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:14.807832 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 4801
2026-09-24T12:46:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3546 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:14 dist-test-slave-9tm1 krb5kdc[5306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:14 dist-test-slave-9tm1 krb5kdc[5306](info): set up 2 sockets
Sep 24 12:46:14 dist-test-slave-9tm1 krb5kdc[5306](info): commencing operation
krb5kdc: starting...
W20260924 12:46:15.809340 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:15 dist-test-slave-9tm1 krb5kdc[5306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:16.013972 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41127
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:16.344383 5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:16.344700 5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:16.344815 5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:16.352496 5319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:16.352630 5319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:16.352677 5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:16.352753 5319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:16.352807 5319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:16.363529 5319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:16.364887 5319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:16.366255 5319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:16.378077 5327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:16.378114 5325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:16.378278 5324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:16.379345 5319 server_base.cc:1034] running on GCE node
I20260924 12:46:16.380223 5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:16.381798 5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:16.383028 5319 hybrid_clock.cc:648] HybridClock initialized: now 1790253976382977 us; error 45 us; skew 500 ppm
I20260924 12:46:16.383389 5319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:16 dist-test-slave-9tm1 krb5kdc[5306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253976, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:16.390588 5319 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:16.392320 5319 webserver.cc:460] Webserver started at http://127.2.81.126:34203/ using document root <none> and password file <none>
I20260924 12:46:16.393095 5319 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:16.393229 5319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:16.393543 5319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:16.396658 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "1e422cca5566446d905dd0bbf831ef7e"
format_stamp: "Formatted at 2026-09-24 12:46:16 on dist-test-slave-9tm1"
server_key: "260cb3c5f93d9e847fc517f9349ecff8"
server_key_iv: "10ff1d4e8442a974352dd049b2c8271f"
server_key_version: "encryptionkey@0"
I20260924 12:46:16.397502 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "1e422cca5566446d905dd0bbf831ef7e"
format_stamp: "Formatted at 2026-09-24 12:46:16 on dist-test-slave-9tm1"
server_key: "260cb3c5f93d9e847fc517f9349ecff8"
server_key_iv: "10ff1d4e8442a974352dd049b2c8271f"
server_key_version: "encryptionkey@0"
I20260924 12:46:16.402947 5319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:46:16.406730 5334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:16.408164 5319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:46:16.408339 5319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "1e422cca5566446d905dd0bbf831ef7e"
format_stamp: "Formatted at 2026-09-24 12:46:16 on dist-test-slave-9tm1"
server_key: "260cb3c5f93d9e847fc517f9349ecff8"
server_key_iv: "10ff1d4e8442a974352dd049b2c8271f"
server_key_version: "encryptionkey@0"
I20260924 12:46:16.408541 5319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:16.429912 5319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:16.434376 5319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:16.434659 5319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:16.452963 5319 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:41127
I20260924 12:46:16.452965 5385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:41127 every 8 connection(s)
I20260924 12:46:16.454447 5319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:16.458220 5386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:16.462332 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5319
I20260924 12:46:16.462550 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:16.463287 2373 external_mini_cluster.cc:1428] Setting key 0c2699efd317b4ae55ef3dd31eb4e5d2
I20260924 12:46:16.471848 5386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e: Bootstrap starting.
I20260924 12:46:16.475980 5386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:16.477427 5386 log.cc:824] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:46:16 dist-test-slave-9tm1 krb5kdc[5306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:16.480993 5386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e: No bootstrap required, opened a new log
I20260924 12:46:16.488792 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:16.465261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.2.81.126:41127)
0924 12:46:16.465990 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:16.466001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:16.466042 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:46:16.467638 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:16.467663 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:16.468126 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:16.468495 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:16.470418 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:16.471819 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:16.473710 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:16.474608 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:16.474783 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:16.479506 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:16.479519 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:16.479540 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:16.479588 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:16.483569 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:16.484439 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:16.484452 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:16.484463 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:16.484699 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:16.485237 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:16.485243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:16.485246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:16.486028 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:16.486338 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:16.486809 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 12:46:16.487320 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":328,"threads_started":1}
I20260924 12:46:16.491309 5386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e422cca5566446d905dd0bbf831ef7e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41127 } }
I20260924 12:46:16.491828 5386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:16.491938 5386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e422cca5566446d905dd0bbf831ef7e, State: Initialized, Role: FOLLOWER
I20260924 12:46:16.492571 5386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e422cca5566446d905dd0bbf831ef7e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41127 } }
I20260924 12:46:16.492797 5386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:16.492954 5386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:16.493117 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:46:16.493141 5386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:16.495285 5386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e422cca5566446d905dd0bbf831ef7e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41127 } }
I20260924 12:46:16.495821 5386 leader_election.cc:302] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e422cca5566446d905dd0bbf831ef7e; no voters:
I20260924 12:46:16.496416 5386 leader_election.cc:288] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:16.496578 5391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:16.497319 5391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [term 1 LEADER]: Becoming Leader. State: Replica: 1e422cca5566446d905dd0bbf831ef7e, State: Running, Role: LEADER
I20260924 12:46:16.498054 5391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e422cca5566446d905dd0bbf831ef7e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41127 } }
I20260924 12:46:16.500258 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e422cca5566446d905dd0bbf831ef7e. Latest consensus state: current_term: 1 leader_uuid: "1e422cca5566446d905dd0bbf831ef7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e422cca5566446d905dd0bbf831ef7e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41127 } } }
I20260924 12:46:16.500666 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:16.501066 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e422cca5566446d905dd0bbf831ef7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e422cca5566446d905dd0bbf831ef7e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41127 } } }
I20260924 12:46:16.501369 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:16.501866 5386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:16.510797 5396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:16.517144 5396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:16.534710 5410 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 12:46:16.538861 5396 catalog_manager.cc:1347] Generated new cluster ID: 9d47e37aeb474985aa7d7bfa735d5ba9
I20260924 12:46:16.539000 5396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:16.548647 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:16.541113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.2.81.126:41127)
0924 12:46:16.541313 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:16.541320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:16.541849 (+ 529us) server_negotiation.cc:407] Connection header received
0924 12:46:16.542018 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:16.542023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:16.542088 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:16.542176 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:16.543191 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:16.543924 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:16.545333 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:16.545831 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:16.545941 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:16.546997 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:16.547002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:16.547005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:16.547706 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:16.547797 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:16.547826 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:16.547936 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:16.548149 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 12:46:16.548243 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260924 12:46:16.550084 5396 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:16.551331 5396 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:16.559672 5396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e422cca5566446d905dd0bbf831ef7e: Generated new TSK 0
I20260924 12:46:16.560652 5396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:16.571656 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5319
I20260924 12:46:16.572942 5381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:16.669076 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5319
2026-09-24T12:46:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1856 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:16 dist-test-slave-9tm1 krb5kdc[5420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:16 dist-test-slave-9tm1 krb5kdc[5420](info): set up 2 sockets
Sep 24 12:46:16 dist-test-slave-9tm1 krb5kdc[5420](info): commencing operation
krb5kdc: starting...
W20260924 12:46:17.673599 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:17 dist-test-slave-9tm1 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:17.869141 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37775
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:18.207772 5433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:18.208124 5433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:18.208297 5433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:18.216583 5433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:18.216717 5433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:18.216789 5433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:18.216868 5433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:18.216920 5433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:18.227252 5433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:18.228821 5433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:18.230217 5433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:18.243496 5439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:18.243505 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:18.244093 5438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:18.244854 5433 server_base.cc:1034] running on GCE node
I20260924 12:46:18.245757 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:18.247388 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:18.248700 5433 hybrid_clock.cc:648] HybridClock initialized: now 1790253978248635 us; error 55 us; skew 500 ppm
I20260924 12:46:18.249122 5433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:18 dist-test-slave-9tm1 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253978, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:18.253091 5433 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:18.254768 5433 webserver.cc:460] Webserver started at http://127.2.81.126:42733/ using document root <none> and password file <none>
I20260924 12:46:18.255498 5433 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:18.255613 5433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:18.255874 5433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:18.259196 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "84ae5974b6284659a2b6c0b3bf7ca464"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-9tm1"
server_key: "2db388485ff5d04910f3f09f63527ba2"
server_key_iv: "4c4282759eba7dfe95ec34a2938961b3"
server_key_version: "encryptionkey@0"
I20260924 12:46:18.260051 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "84ae5974b6284659a2b6c0b3bf7ca464"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-9tm1"
server_key: "2db388485ff5d04910f3f09f63527ba2"
server_key_iv: "4c4282759eba7dfe95ec34a2938961b3"
server_key_version: "encryptionkey@0"
I20260924 12:46:18.266009 5433 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 12:46:18.269977 5448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:18.271466 5433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:18.271647 5433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "84ae5974b6284659a2b6c0b3bf7ca464"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-9tm1"
server_key: "2db388485ff5d04910f3f09f63527ba2"
server_key_iv: "4c4282759eba7dfe95ec34a2938961b3"
server_key_version: "encryptionkey@0"
I20260924 12:46:18.271847 5433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:18.295001 5433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:18.300348 5433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:18.300644 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:18.319573 5433 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:37775
I20260924 12:46:18.319618 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:37775 every 8 connection(s)
I20260924 12:46:18.320896 5433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:18.324683 5500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:18.329026 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5433
I20260924 12:46:18.329258 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:18.330025 2373 external_mini_cluster.cc:1428] Setting key 0799a26275dffa633ad9dab549785188
I20260924 12:46:18.337407 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464: Bootstrap starting.
I20260924 12:46:18.341298 5500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:18.342541 5500 log.cc:824] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:46:18 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:18.346659 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464: No bootstrap required, opened a new log
I20260924 12:46:18.356256 5500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37775 } }
I20260924 12:46:18.356665 5500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:18.356757 5500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ae5974b6284659a2b6c0b3bf7ca464, State: Initialized, Role: FOLLOWER
I20260924 12:46:18.356834 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:18.332256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.2.81.126:37775)
0924 12:46:18.333072 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:18.333085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:18.333131 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:46:18.335279 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:18.335307 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:18.335779 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:18.336144 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:18.338040 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:18.339484 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:18.341305 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:18.342132 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:18.342297 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:18.347634 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:18.347643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:18.347662 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:18.347704 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:18.351364 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:18.352653 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:18.352663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:18.352671 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:18.353222 (+ 551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:18.353725 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:18.353729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:18.353731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:18.354373 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:18.354673 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:18.355193 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 12:46:18.355754 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":354,"threads_started":1}
I20260924 12:46:18.357419 5500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37775 } }
I20260924 12:46:18.357654 5500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:18.357776 5500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:18.357936 5500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:18.360736 5500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37775 } }
I20260924 12:46:18.361315 5500 leader_election.cc:302] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ae5974b6284659a2b6c0b3bf7ca464; no voters:
I20260924 12:46:18.361863 5500 leader_election.cc:288] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:18.361986 5505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:18.362591 5505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [term 1 LEADER]: Becoming Leader. State: Replica: 84ae5974b6284659a2b6c0b3bf7ca464, State: Running, Role: LEADER
I20260924 12:46:18.363178 5505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37775 } }
I20260924 12:46:18.363988 5500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:18.365758 5506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37775 } } }
I20260924 12:46:18.366206 5506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:18.367070 5512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:18.367198 5507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ae5974b6284659a2b6c0b3bf7ca464. Latest consensus state: current_term: 1 leader_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae5974b6284659a2b6c0b3bf7ca464" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37775 } } }
I20260924 12:46:18.368599 5507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:18.372677 5512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:18.381629 5512 catalog_manager.cc:1347] Generated new cluster ID: f540ed7a201b45759544c40e056a1b20
I20260924 12:46:18.381758 5512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:18.398459 5512 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:18.400038 5512 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:18.408602 5512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ae5974b6284659a2b6c0b3bf7ca464: Generated new TSK 0
I20260924 12:46:18.409807 5512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:46:18.474642 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37775
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:18.814034 5526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:18.814322 5526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:18.814476 5526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:18.821939 5526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:18.822081 5526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:18.822256 5526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:18.832131 5526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:18.833501 5526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:18.834779 5526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:18.847604 5531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:18.847652 5534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:18.847599 5532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:18.848351 5526 server_base.cc:1034] running on GCE node
I20260924 12:46:18.849027 5526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:18.849990 5526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:18.851217 5526 hybrid_clock.cc:648] HybridClock initialized: now 1790253978851158 us; error 50 us; skew 500 ppm
I20260924 12:46:18.851616 5526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:18 dist-test-slave-9tm1 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253978, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:18.854830 5526 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:46:18.856302 5526 webserver.cc:460] Webserver started at http://127.2.81.65:41971/ using document root <none> and password file <none>
I20260924 12:46:18.857002 5526 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:18.857111 5526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:18.857386 5526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:18.860208 5526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "e5efb772d9724904be633705232beb8e"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-9tm1"
server_key: "71c0ebc7a0219171564f1f2b73d31a73"
server_key_iv: "ea17063efd20ed423820526f388f2e7e"
server_key_version: "encryptionkey@0"
I20260924 12:46:18.861004 5526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "e5efb772d9724904be633705232beb8e"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-9tm1"
server_key: "71c0ebc7a0219171564f1f2b73d31a73"
server_key_iv: "ea17063efd20ed423820526f388f2e7e"
server_key_version: "encryptionkey@0"
I20260924 12:46:18.866068 5526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:46:18.869730 5541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:18.871168 5526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:18.871326 5526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "e5efb772d9724904be633705232beb8e"
format_stamp: "Formatted at 2026-09-24 12:46:18 on dist-test-slave-9tm1"
server_key: "71c0ebc7a0219171564f1f2b73d31a73"
server_key_iv: "ea17063efd20ed423820526f388f2e7e"
server_key_version: "encryptionkey@0"
I20260924 12:46:18.871536 5526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:18.892294 5526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:18.896638 5526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:18.896915 5526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:18.897892 5526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:18.899222 5526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:18.899298 5526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:18.899403 5526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:18.899454 5526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:18.929358 5526 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:43043
I20260924 12:46:18.929414 5653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:43043 every 8 connection(s)
I20260924 12:46:18.930626 5526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:18.935015 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5526
I20260924 12:46:18.935209 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:18.935788 2373 external_mini_cluster.cc:1428] Setting key 5beac1ed8a0bbb5b7c65350159f93059
Sep 24 12:46:18 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253978, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:18.952775 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:18.933130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:39899 (local address 127.2.81.126:37775)
0924 12:46:18.933558 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:18.933563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:18.934428 (+ 865us) server_negotiation.cc:407] Connection header received
0924 12:46:18.936345 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:18.936349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:18.936418 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:18.936524 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:18.938391 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:18.939114 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:18.941316 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:18.941748 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:18.941854 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:18.946349 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:18.946355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:18.946358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:18.946394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:18.948913 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:18.950133 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:18.950138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:18.950140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:18.950276 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:18.950872 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:18.950876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:18.950878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:18.951170 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:18.951292 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:18.952285 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 12:46:18.952472 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":230,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:46:18.954228 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:18.933562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37775 (local address 127.2.81.65:39899)
0924 12:46:18.934261 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:18.934303 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:18.936050 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:18.936802 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:18.936811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:18.937371 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:18.938175 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:18.938203 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:18.939286 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:18.939292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:18.941130 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:18.941146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:18.941975 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:18.941981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:18.942265 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:18.943045 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:18.943078 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:18.946008 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:18.949229 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:18.949239 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:18.949259 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:18.949921 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:18.950405 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:18.950410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:18.950414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:18.950695 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:18.951309 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:18.951318 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:18.951821 (+ 503us) client_negotiation.cc:769] Sending connection context
0924 12:46:18.952147 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 12:46:18.952725 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":260,"threads_started":1}
I20260924 12:46:18.955914 5654 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37775
I20260924 12:46:18.956245 5654 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:46:18.957091 5654 heartbeater.cc:511] Master 127.2.81.126:37775 requested a full tablet report, sending...
I20260924 12:46:18.959473 5465 ts_manager.cc:194] Registered new tserver with Master: e5efb772d9724904be633705232beb8e (127.2.81.65:43043)
I20260924 12:46:18.960951 5465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:39899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:18.981638 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37775
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:19.334264 5660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:19.334681 5660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:19.334945 5660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:19.342883 5660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:19.343020 5660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:19.343168 5660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:19.353740 5660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:19.355079 5660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:19.356374 5660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:19.369822 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.369773 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.369844 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:19.370713 5660 server_base.cc:1034] running on GCE node
I20260924 12:46:19.371306 5660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:19.372251 5660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:19.373454 5660 hybrid_clock.cc:648] HybridClock initialized: now 1790253979373389 us; error 49 us; skew 500 ppm
I20260924 12:46:19.373854 5660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:19 dist-test-slave-9tm1 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253979, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:19.377302 5660 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:19.378926 5660 webserver.cc:460] Webserver started at http://127.2.81.66:34089/ using document root <none> and password file <none>
I20260924 12:46:19.379659 5660 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:19.379781 5660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:19.380082 5660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:19.383559 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "ee6e117d59e04731b6c1824aa72a4664"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-9tm1"
server_key: "3df02d560d4862358ed1851029ee21d8"
server_key_iv: "5ffd57f5e203dbffe17dfb0362eced87"
server_key_version: "encryptionkey@0"
I20260924 12:46:19.384469 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "ee6e117d59e04731b6c1824aa72a4664"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-9tm1"
server_key: "3df02d560d4862358ed1851029ee21d8"
server_key_iv: "5ffd57f5e203dbffe17dfb0362eced87"
server_key_version: "encryptionkey@0"
I20260924 12:46:19.390242 5660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:46:19.394387 5675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.396113 5660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260924 12:46:19.396397 5660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "ee6e117d59e04731b6c1824aa72a4664"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-9tm1"
server_key: "3df02d560d4862358ed1851029ee21d8"
server_key_iv: "5ffd57f5e203dbffe17dfb0362eced87"
server_key_version: "encryptionkey@0"
I20260924 12:46:19.396726 5660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:19.417598 5660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:19.423391 5660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:19.423723 5660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:19.424749 5660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:19.426213 5660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:19.426295 5660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.426419 5660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:19.426473 5660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.459081 5660 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:46229
I20260924 12:46:19.459180 5787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:46229 every 8 connection(s)
I20260924 12:46:19.460606 5660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:19.463981 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5660
I20260924 12:46:19.464167 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:19.464709 2373 external_mini_cluster.cc:1428] Setting key 17da077c2762481fa4fbaf3a03c40bf2
Sep 24 12:46:19 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253979, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:46:19.484356 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:19.463138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:42587 (local address 127.2.81.126:37775)
0924 12:46:19.463591 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:19.463596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:19.465626 (+ 2030us) server_negotiation.cc:407] Connection header received
0924 12:46:19.466643 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:19.466648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:19.466720 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:19.466819 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:19.469091 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.470028 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.472345 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.473006 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.473151 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:19.477347 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:19.477353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:19.477358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:19.477400 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:19.480849 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:19.481851 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:19.481858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:19.481862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:19.482029 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:19.482500 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:19.482506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:19.482510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:19.482877 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:19.483029 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:19.483801 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 12:46:19.484043 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":223,"threads_started":1}
I20260924 12:46:19.485519 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:19.463607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37775 (local address 127.2.81.66:42587)
0924 12:46:19.464420 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:19.464457 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:19.466389 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:19.467131 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:19.467143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:19.467901 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:19.468801 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:19.468830 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.470240 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.470246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:19.472157 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:19.472175 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.473246 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.473252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:19.473478 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:19.474209 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:19.474244 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:19.477062 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:19.481138 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:19.481148 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:19.481164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:19.481696 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:19.482164 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:19.482167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:19.482170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:19.482371 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:19.483024 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:19.483031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:19.483414 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 12:46:19.483662 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 12:46:19.484204 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":347,"threads_started":1}
I20260924 12:46:19.487136 5788 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:46:19.487461 5788 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:19.488337 5788 heartbeater.cc:511] Master 127.2.81.126:37775 requested a full tablet report, sending...
I20260924 12:46:19.490021 5465 ts_manager.cc:194] Registered new tserver with Master: ee6e117d59e04731b6c1824aa72a4664 (127.2.81.66:46229)
I20260924 12:46:19.491061 5465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:42587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:19.513947 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37775
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:19.845234 5794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:19.845516 5794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:19.845608 5794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:19.853268 5794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:19.853397 5794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:19.853528 5794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:19.863508 5794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:19.864764 5794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:19.866052 5794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:19.878791 5800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.878988 5802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:19.878858 5799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:19.880056 5794 server_base.cc:1034] running on GCE node
I20260924 12:46:19.880792 5794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:19.881796 5794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:19.883008 5794 hybrid_clock.cc:648] HybridClock initialized: now 1790253979882956 us; error 39 us; skew 500 ppm
I20260924 12:46:19.883314 5794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:19 dist-test-slave-9tm1 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253979, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:19.886441 5794 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:19.887959 5794 webserver.cc:460] Webserver started at http://127.2.81.67:33491/ using document root <none> and password file <none>
I20260924 12:46:19.888652 5794 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:19.888767 5794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:19.889096 5794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:19.892203 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "997f8427bea64dc0a394fbe2dfe04fff"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-9tm1"
server_key: "3a307b8040fcece929f9e45adaf4ac1f"
server_key_iv: "f8adfd54424ff27f6f9f578245b9100b"
server_key_version: "encryptionkey@0"
I20260924 12:46:19.893105 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "997f8427bea64dc0a394fbe2dfe04fff"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-9tm1"
server_key: "3a307b8040fcece929f9e45adaf4ac1f"
server_key_iv: "f8adfd54424ff27f6f9f578245b9100b"
server_key_version: "encryptionkey@0"
I20260924 12:46:19.898617 5794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 12:46:19.902345 5809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.903800 5794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:19.903968 5794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "997f8427bea64dc0a394fbe2dfe04fff"
format_stamp: "Formatted at 2026-09-24 12:46:19 on dist-test-slave-9tm1"
server_key: "3a307b8040fcece929f9e45adaf4ac1f"
server_key_iv: "f8adfd54424ff27f6f9f578245b9100b"
server_key_version: "encryptionkey@0"
I20260924 12:46:19.904182 5794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:19.917860 5794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:19.921901 5794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:19.922168 5794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:19.923111 5794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:19.924394 5794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:19.924465 5794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.924569 5794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:19.924621 5794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:19.956153 5794 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:37529
I20260924 12:46:19.956198 5921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:37529 every 8 connection(s)
I20260924 12:46:19.957754 5794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:19.961797 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5794
I20260924 12:46:19.961977 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:19.962401 2373 external_mini_cluster.cc:1428] Setting key 101a51aa6ad6c6c303d3ce70f0de8635
I20260924 12:46:19.964711 5654 heartbeater.cc:503] Master 127.2.81.126:37775 was elected leader, sending a full tablet report...
Sep 24 12:46:19 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253979, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:19.978124 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:19.960314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:42629 (local address 127.2.81.126:37775)
0924 12:46:19.960488 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:19.960492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:19.961756 (+ 1264us) server_negotiation.cc:407] Connection header received
0924 12:46:19.963819 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:19.963824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:19.963920 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:19.964014 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:19.966002 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.966709 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.968339 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.968710 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.968817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:19.972455 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:19.972459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:19.972463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:19.972495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:19.974917 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:19.975655 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:19.975669 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:19.975672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:19.975851 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:19.976266 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:19.976271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:19.976274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:19.976543 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:19.976686 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:19.977575 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 12:46:19.977795 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:46:19.979070 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:19.960772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37775 (local address 127.2.81.67:42629)
0924 12:46:19.961588 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:19.961629 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:19.963518 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:19.964327 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:19.964337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:19.965107 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:19.965817 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:19.965836 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.966851 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.966856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:19.968203 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:19.968216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:19.968939 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:19.968946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:19.969214 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:19.970057 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:19.970080 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:19.972242 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:19.975065 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:19.975074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:19.975092 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:19.975532 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:19.975954 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:19.975960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:19.975963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:19.976154 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:19.976682 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:19.976691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:19.977169 (+ 478us) client_negotiation.cc:769] Sending connection context
0924 12:46:19.977456 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 12:46:19.977997 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":328,"threads_started":1}
I20260924 12:46:19.980472 5922 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37775
I20260924 12:46:19.980782 5922 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:19.981611 5922 heartbeater.cc:511] Master 127.2.81.126:37775 requested a full tablet report, sending...
I20260924 12:46:19.983075 5465 ts_manager.cc:194] Registered new tserver with Master: 997f8427bea64dc0a394fbe2dfe04fff (127.2.81.67:37529)
I20260924 12:46:19.983903 5465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:42629
I20260924 12:46:19.990068 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:20.036962 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:20.022484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.2.81.126:37775)
0924 12:46:20.022682 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:20.022687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:20.023080 (+ 393us) server_negotiation.cc:407] Connection header received
0924 12:46:20.023225 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:20.023229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:20.023303 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:20.023399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:20.024917 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.025730 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.027052 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.027457 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.027568 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:20.031520 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:20.031526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:20.031531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:20.031609 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:20.034246 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.034940 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.034945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.034947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.035093 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.035592 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.035596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.035599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.035948 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:20.036162 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:20.036486 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 12:46:20.036706 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 12:46:20.037676 5465 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53768
I20260924 12:46:20.039577 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5526
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:20.057940 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:20.040667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.2.81.65:43043)
0924 12:46:20.041276 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:20.041283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:20.041308 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:46:20.041362 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:20.041367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:20.041569 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:20.041707 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:20.042808 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.043854 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.045733 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.046345 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.046468 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:20.051086 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:20.051104 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:20.051121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:20.051189 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:20.054260 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.055319 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.055327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.055330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.055480 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.055954 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.055958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.055960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.056359 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:20.056865 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:20.057222 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 12:46:20.057479 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":324,"threads_started":1}
W20260924 12:46:20.059332 5649 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52836
W20260924 12:46:20.060922 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5526: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:20.061206 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5526
I20260924 12:46:20.073261 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5660
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:46:20.089740 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:20.074038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.2.81.66:46229)
0924 12:46:20.074573 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:20.074580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:20.074606 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:46:20.074674 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:20.074680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:20.074875 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:20.075021 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:20.076189 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.077319 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.078637 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.079207 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.079311 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:20.083664 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:20.083678 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:20.083693 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:20.083731 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:20.086425 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.087431 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.087441 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.087443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.087595 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.088179 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.088184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.088186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.088552 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:20.088829 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:20.089201 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 12:46:20.089417 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
W20260924 12:46:20.091290 5783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46536
W20260924 12:46:20.092811 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5660: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:20.092967 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5660
I20260924 12:46:20.104887 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5794
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:46:20.122133 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:20.106004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.2.81.67:37529)
0924 12:46:20.106643 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:20.106650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:20.106679 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:46:20.106738 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:20.106745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:20.106960 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:20.107150 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:20.108513 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.109858 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.111129 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:20.111710 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:20.111828 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:20.116266 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:20.116281 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:20.116296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:20.116337 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:20.119186 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.119930 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.119938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.119940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.120071 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:20.120422 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:20.120426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:20.120428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:20.120780 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:20.121103 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:20.121511 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 12:46:20.121781 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":353,"threads_started":1}
W20260924 12:46:20.123476 5917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40230
W20260924 12:46:20.124742 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:46:20.124910 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5794
I20260924 12:46:20.136500 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5433
I20260924 12:46:20.137596 5495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:20.237845 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5433
2026-09-24T12:46:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3573 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5939](info): set up 2 sockets
Sep 24 12:46:20 dist-test-slave-9tm1 krb5kdc[5939](info): commencing operation
krb5kdc: starting...
W20260924 12:46:21.245278 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:21 dist-test-slave-9tm1 krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:21.438941 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:38253
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:38253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:21.784116 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:21.784435 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:21.784554 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:21.792002 5952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:21.792133 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:21.792181 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:21.792258 5952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:21.792316 5952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:21.802367 5952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:38253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:21.803689 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:21.805043 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:21.815853 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:21.816566 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:21.817548 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:21.818272 5952 server_base.cc:1034] running on GCE node
I20260924 12:46:21.819056 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:21.820462 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:21.821672 5952 hybrid_clock.cc:648] HybridClock initialized: now 1790253981821621 us; error 35 us; skew 500 ppm
I20260924 12:46:21.822013 5952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:21 dist-test-slave-9tm1 krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253981, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:21.825042 5952 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:21.826630 5952 webserver.cc:460] Webserver started at http://127.2.81.126:37415/ using document root <none> and password file <none>
I20260924 12:46:21.827319 5952 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:21.827432 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:21.827723 5952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:21.830768 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "17ccff1211f2443bb4e016f391bc6cab"
format_stamp: "Formatted at 2026-09-24 12:46:21 on dist-test-slave-9tm1"
server_key: "7f0f455b6460976c9ba38053cee976fe"
server_key_iv: "886c0d0f63631753501efb016254806b"
server_key_version: "encryptionkey@0"
I20260924 12:46:21.831566 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "17ccff1211f2443bb4e016f391bc6cab"
format_stamp: "Formatted at 2026-09-24 12:46:21 on dist-test-slave-9tm1"
server_key: "7f0f455b6460976c9ba38053cee976fe"
server_key_iv: "886c0d0f63631753501efb016254806b"
server_key_version: "encryptionkey@0"
I20260924 12:46:21.836863 5952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:46:21.840708 5967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:21.842243 5952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:46:21.842424 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "17ccff1211f2443bb4e016f391bc6cab"
format_stamp: "Formatted at 2026-09-24 12:46:21 on dist-test-slave-9tm1"
server_key: "7f0f455b6460976c9ba38053cee976fe"
server_key_iv: "886c0d0f63631753501efb016254806b"
server_key_version: "encryptionkey@0"
I20260924 12:46:21.842628 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:21.867630 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:21.871726 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:21.872010 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:21.889796 5952 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:38253
I20260924 12:46:21.889834 6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:38253 every 8 connection(s)
I20260924 12:46:21.891186 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:21.895143 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:21.897598 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 5952
I20260924 12:46:21.897811 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:21.898437 2373 external_mini_cluster.cc:1428] Setting key 55256f714e4abd46b189aa79e4c35cd4
I20260924 12:46:21.907320 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab: Bootstrap starting.
I20260924 12:46:21.911089 6019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:21.912359 6019 log.cc:824] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:46:21 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:21.915222 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab: No bootstrap required, opened a new log
I20260924 12:46:21.921051 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:21.900491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.2.81.126:38253)
0924 12:46:21.901291 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:21.901304 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:21.901345 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:46:21.902995 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:21.903020 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:21.903412 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:21.903762 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:21.905583 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:21.906961 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:21.908607 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:21.909352 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:21.909504 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:21.914014 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:21.914022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:21.914038 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:21.914075 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:21.916949 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:21.917657 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:21.917664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:21.917671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:21.917859 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:21.918235 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:21.918240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:21.918244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:21.918776 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:21.919054 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:21.919499 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 12:46:21.919932 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":384,"threads_started":1}
I20260924 12:46:21.923271 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ccff1211f2443bb4e016f391bc6cab" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38253 } }
I20260924 12:46:21.923628 6019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:21.923722 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ccff1211f2443bb4e016f391bc6cab, State: Initialized, Role: FOLLOWER
I20260924 12:46:21.924286 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ccff1211f2443bb4e016f391bc6cab" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38253 } }
I20260924 12:46:21.924480 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:21.924580 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:21.924695 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:21.926766 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ccff1211f2443bb4e016f391bc6cab" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38253 } }
I20260924 12:46:21.927278 6019 leader_election.cc:302] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17ccff1211f2443bb4e016f391bc6cab; no voters:
I20260924 12:46:21.927888 6019 leader_election.cc:288] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:21.928015 6024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:21.928632 6024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [term 1 LEADER]: Becoming Leader. State: Replica: 17ccff1211f2443bb4e016f391bc6cab, State: Running, Role: LEADER
I20260924 12:46:21.929205 6024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ccff1211f2443bb4e016f391bc6cab" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38253 } }
I20260924 12:46:21.929976 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:21.931665 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17ccff1211f2443bb4e016f391bc6cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ccff1211f2443bb4e016f391bc6cab" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38253 } } }
I20260924 12:46:21.931977 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:21.932777 6029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:21.932621 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17ccff1211f2443bb4e016f391bc6cab. Latest consensus state: current_term: 1 leader_uuid: "17ccff1211f2443bb4e016f391bc6cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ccff1211f2443bb4e016f391bc6cab" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38253 } } }
I20260924 12:46:21.932941 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:21.939041 6029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:21.948499 6029 catalog_manager.cc:1347] Generated new cluster ID: 0b3092faab194e88828bc832a29af5cf
I20260924 12:46:21.948629 6029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:21.972697 6029 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:21.974730 6029 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:21.987754 6029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17ccff1211f2443bb4e016f391bc6cab: Generated new TSK 0
I20260924 12:46:21.989027 6029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:46:22.034906 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38253
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:22.363343 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:22.363642 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:22.363737 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:22.370903 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:22.371035 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:22.371162 6045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:22.381377 6045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:22.382604 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:22.383875 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:22.396454 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:22.396471 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:22.396703 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:22.397265 6045 server_base.cc:1034] running on GCE node
I20260924 12:46:22.397872 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:22.398833 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:22.400041 6045 hybrid_clock.cc:648] HybridClock initialized: now 1790253982399986 us; error 51 us; skew 500 ppm
I20260924 12:46:22.400352 6045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:22 dist-test-slave-9tm1 krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:22.403396 6045 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:46:22.404860 6045 webserver.cc:460] Webserver started at http://127.2.81.65:33377/ using document root <none> and password file <none>
I20260924 12:46:22.405578 6045 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:22.405699 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:22.405994 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:22.409242 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "ed2b92e678a7404a9d1814b0e303812e"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-9tm1"
server_key: "eb6c4d49ddf95e14769d7aa8d2ca3a10"
server_key_iv: "a039e267cc9f461b43efda3b9ed3ae4c"
server_key_version: "encryptionkey@0"
I20260924 12:46:22.410027 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "ed2b92e678a7404a9d1814b0e303812e"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-9tm1"
server_key: "eb6c4d49ddf95e14769d7aa8d2ca3a10"
server_key_iv: "a039e267cc9f461b43efda3b9ed3ae4c"
server_key_version: "encryptionkey@0"
I20260924 12:46:22.415195 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:22.418988 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:22.420315 6045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:22.420459 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "ed2b92e678a7404a9d1814b0e303812e"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-9tm1"
server_key: "eb6c4d49ddf95e14769d7aa8d2ca3a10"
server_key_iv: "a039e267cc9f461b43efda3b9ed3ae4c"
server_key_version: "encryptionkey@0"
I20260924 12:46:22.420646 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:22.457238 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:22.461679 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:22.462193 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:22.463341 6045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:22.464751 6045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:22.464830 6045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:22.464959 6045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:22.465029 6045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:46:22.496961 6045 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:39357
I20260924 12:46:22.497022 6172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:39357 every 8 connection(s)
I20260924 12:46:22.498288 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:22.503544 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6045
I20260924 12:46:22.503727 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:22.504352 2373 external_mini_cluster.cc:1428] Setting key c1466763f7d3743e5cb75082f8e0103a
Sep 24 12:46:22 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:22.519727 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:22.500789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:55721 (local address 127.2.81.126:38253)
0924 12:46:22.501244 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:22.501250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:22.502236 (+ 986us) server_negotiation.cc:407] Connection header received
0924 12:46:22.504109 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:22.504115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:22.504198 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:22.504296 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:22.506295 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:22.507314 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:22.509040 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:22.509598 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:22.509727 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:22.513713 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:22.513719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:22.513722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:22.513758 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:22.516391 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:22.517362 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:22.517366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:22.517369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:22.517492 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:22.517934 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:22.517937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:22.517939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:22.518213 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:22.518345 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:22.519101 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 12:46:22.519381 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260924 12:46:22.520666 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:22.501374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:38253 (local address 127.2.81.65:55721)
0924 12:46:22.502079 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:22.502122 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:22.503831 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:22.504571 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:22.504580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:22.505273 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:22.506079 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:22.506107 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:22.507562 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:22.507567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:22.508857 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:22.508870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:22.509837 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:22.509843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:22.510064 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:22.510726 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:22.510755 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:22.513409 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:22.516639 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:22.516651 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:22.516671 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:22.517223 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:22.517601 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:22.517606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:22.517610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:22.517812 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:22.518336 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:22.518344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:22.518707 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 12:46:22.518949 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 12:46:22.519487 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":291,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:46:22.522192 6173 heartbeater.cc:347] Connected to a master server at 127.2.81.126:38253
I20260924 12:46:22.522465 6173 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:22.523181 6173 heartbeater.cc:511] Master 127.2.81.126:38253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:46:22.525439 5984 ts_manager.cc:194] Registered new tserver with Master: ed2b92e678a7404a9d1814b0e303812e (127.2.81.65:39357)
I20260924 12:46:22.526886 5984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:55721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:22.549316 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38253
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:22.880061 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:22.880363 6179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:22.880457 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:22.887851 6179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:22.887979 6179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:22.888099 6179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:22.898576 6179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:22.899935 6179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:22.901311 6179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:22.915712 6187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:22.915712 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:22.916281 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:22.916510 6179 server_base.cc:1034] running on GCE node
I20260924 12:46:22.917308 6179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:22.918391 6179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:22.919672 6179 hybrid_clock.cc:648] HybridClock initialized: now 1790253982919558 us; error 69 us; skew 500 ppm
I20260924 12:46:22.920163 6179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:22 dist-test-slave-9tm1 krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:22.923410 6179 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:22.924966 6179 webserver.cc:460] Webserver started at http://127.2.81.66:39021/ using document root <none> and password file <none>
I20260924 12:46:22.925689 6179 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:22.925797 6179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:22.926084 6179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:22.929144 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "75a699bc0fe24f00ad4e7a9754667796"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-9tm1"
server_key: "5c66f1822b806dd7da7515aa1bf59953"
server_key_iv: "605257231d830d5856abc389835067b7"
server_key_version: "encryptionkey@0"
I20260924 12:46:22.929908 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "75a699bc0fe24f00ad4e7a9754667796"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-9tm1"
server_key: "5c66f1822b806dd7da7515aa1bf59953"
server_key_iv: "605257231d830d5856abc389835067b7"
server_key_version: "encryptionkey@0"
I20260924 12:46:22.935254 6179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:22.938869 6194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:22.940289 6179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:46:22.940428 6179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "75a699bc0fe24f00ad4e7a9754667796"
format_stamp: "Formatted at 2026-09-24 12:46:22 on dist-test-slave-9tm1"
server_key: "5c66f1822b806dd7da7515aa1bf59953"
server_key_iv: "605257231d830d5856abc389835067b7"
server_key_version: "encryptionkey@0"
I20260924 12:46:22.940665 6179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:22.973815 6179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:22.978192 6179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:22.978463 6179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:22.979416 6179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:22.980779 6179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:22.980854 6179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:22.980959 6179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:22.981026 6179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:46:23.011881 6179 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:33113
I20260924 12:46:23.011950 6306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:33113 every 8 connection(s)
I20260924 12:46:23.013212 6179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:23.017997 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6179
I20260924 12:46:23.018167 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:23.018554 2373 external_mini_cluster.cc:1428] Setting key 764cdba801aa47fdf05f3f8031dfb379
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253982, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:23.033309 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.015723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:39177 (local address 127.2.81.126:38253)
0924 12:46:23.015888 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.015893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.017031 (+ 1138us) server_negotiation.cc:407] Connection header received
0924 12:46:23.019058 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.019064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.019153 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.019258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.020962 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.022045 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.023696 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.024319 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.024435 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.027783 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.027788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.027791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.027826 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.030345 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.031087 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.031092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.031096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.031236 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.031652 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.031658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.031662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.031916 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.032040 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.032722 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.032939 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 12:46:23.034279 6309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.016144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:38253 (local address 127.2.81.66:39177)
0924 12:46:23.016834 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:23.016878 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:23.018763 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:23.019506 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:23.019516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:23.020048 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:23.020775 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:23.020794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.022223 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.022227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:23.023546 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:23.023560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.024517 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.024522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:23.024730 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.025414 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:23.025436 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:23.027540 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:23.030511 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:23.030520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:23.030534 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:23.030972 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:23.031337 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:23.031342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:23.031346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:23.031543 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:23.032034 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:23.032041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:23.032386 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 12:46:23.032629 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 12:46:23.033189 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":290,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:46:23.035702 6307 heartbeater.cc:347] Connected to a master server at 127.2.81.126:38253
I20260924 12:46:23.036031 6307 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:23.036801 6307 heartbeater.cc:511] Master 127.2.81.126:38253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:46:23.038444 5984 ts_manager.cc:194] Registered new tserver with Master: 75a699bc0fe24f00ad4e7a9754667796 (127.2.81.66:33113)
I20260924 12:46:23.039242 5984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:39177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:23.061321 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38253
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:23.388098 6312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:23.388386 6312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:23.388478 6312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:23.395972 6312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:23.396106 6312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:23.396219 6312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:23.406185 6312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:23.407431 6312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:23.408718 6312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:23.421793 6318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:23.421733 6317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:23.421762 6320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:23.422514 6312 server_base.cc:1034] running on GCE node
I20260924 12:46:23.423169 6312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:23.424216 6312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:23.425419 6312 hybrid_clock.cc:648] HybridClock initialized: now 1790253983425369 us; error 48 us; skew 500 ppm
I20260924 12:46:23.425794 6312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:23.429039 6312 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:23.430729 6312 webserver.cc:460] Webserver started at http://127.2.81.67:33135/ using document root <none> and password file <none>
I20260924 12:46:23.431417 6312 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:23.431535 6312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:23.431826 6312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:23.434921 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "fbc1a274e92e459998577a799fc690a1"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-9tm1"
server_key: "4bca2aefa202c5dd83c63d66de079f6e"
server_key_iv: "e670cb85091dbf839a433878d767fc23"
server_key_version: "encryptionkey@0"
I20260924 12:46:23.435781 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "fbc1a274e92e459998577a799fc690a1"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-9tm1"
server_key: "4bca2aefa202c5dd83c63d66de079f6e"
server_key_iv: "e670cb85091dbf839a433878d767fc23"
server_key_version: "encryptionkey@0"
I20260924 12:46:23.441469 6312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 12:46:23.445328 6327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.446915 6312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:23.447089 6312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "fbc1a274e92e459998577a799fc690a1"
format_stamp: "Formatted at 2026-09-24 12:46:23 on dist-test-slave-9tm1"
server_key: "4bca2aefa202c5dd83c63d66de079f6e"
server_key_iv: "e670cb85091dbf839a433878d767fc23"
server_key_version: "encryptionkey@0"
I20260924 12:46:23.447297 6312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:23.468148 6312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:23.472429 6312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:23.472714 6312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:23.473819 6312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:23.475219 6312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:23.475291 6312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.475404 6312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:23.475453 6312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:23.507436 6312 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:45085
I20260924 12:46:23.507494 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:45085 every 8 connection(s)
I20260924 12:46:23.508656 6312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:23.510318 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6312
I20260924 12:46:23.510489 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:23.511009 2373 external_mini_cluster.cc:1428] Setting key 61e000c58828eff7a9ec174cf42db544
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:23.528543 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.511207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:36641 (local address 127.2.81.126:38253)
0924 12:46:23.511374 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.511381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.512742 (+ 1361us) server_negotiation.cc:407] Connection header received
0924 12:46:23.514632 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.514636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.514762 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.514866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.516392 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.517251 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.518790 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.519194 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.519302 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.522715 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.522721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.522724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.522757 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.525565 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.526329 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.526334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.526337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.526470 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.526873 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.526877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.526879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.527138 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.527275 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.527973 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.528181 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 12:46:23.529574 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.511813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:38253 (local address 127.2.81.67:36641)
0924 12:46:23.512595 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:23.512637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:23.514385 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:23.515110 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:23.515118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:23.515655 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:23.516222 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:23.516240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.517395 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.517399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:23.518657 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:23.518670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.519417 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.519424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:23.519653 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.520258 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:23.520284 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:23.522487 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:23.525710 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:23.525724 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:23.525740 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:23.526213 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:23.526572 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:23.526577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:23.526581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:23.526767 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:23.527272 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:23.527281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:23.527611 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 12:46:23.527869 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 12:46:23.528409 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":321,"threads_started":1}
I20260924 12:46:23.530066 6173 heartbeater.cc:503] Master 127.2.81.126:38253 was elected leader, sending a full tablet report...
I20260924 12:46:23.530973 6440 heartbeater.cc:347] Connected to a master server at 127.2.81.126:38253
I20260924 12:46:23.531286 6440 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:23.532038 6440 heartbeater.cc:511] Master 127.2.81.126:38253 requested a full tablet report, sending...
I20260924 12:46:23.533357 5984 ts_manager.cc:194] Registered new tserver with Master: fbc1a274e92e459998577a799fc690a1 (127.2.81.67:45085)
I20260924 12:46:23.534102 5984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:36641
I20260924 12:46:23.538187 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:23.584403 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.570391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.2.81.65:39357)
0924 12:46:23.570931 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.570938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.570965 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:46:23.571022 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.571028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.571242 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.571378 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.572722 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.573781 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.574943 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.575496 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.575596 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.579230 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.579241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.579255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.579287 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.581976 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.582594 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.582600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.582602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.582721 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.583037 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.583040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.583042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.583372 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.583590 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.583863 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.584051 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":258,"threads_started":1}
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:23.601423 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.588424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.2.81.126:38253)
0924 12:46:23.588637 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.588643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.588871 (+ 228us) server_negotiation.cc:407] Connection header received
0924 12:46:23.589080 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.589084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.589145 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.589229 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.590388 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.591088 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.592287 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.592692 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.592801 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.596521 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.596526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.596530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.596563 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.598966 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.599558 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.599563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.599566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.599696 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.600026 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.600031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.600033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.600359 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.600588 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.600872 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.601124 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 12:46:23.602016 5984 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35812
I20260924 12:46:23.603106 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6045
I20260924 12:46:23.614576 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.603852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.2.81.65:39357)
0924 12:46:23.604048 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.604054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.604221 (+ 167us) server_negotiation.cc:407] Connection header received
0924 12:46:23.604367 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.604370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.604445 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.604521 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.605637 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.606350 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.607447 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.607853 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.607947 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.609982 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.609987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.609991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.610021 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.612433 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.613014 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.613019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.613021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.613138 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.613432 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.613435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.613437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.613663 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.613823 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.614053 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.614264 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:23.615159 6168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:23.698875 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6045
I20260924 12:46:23.711925 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6179
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:46:23.727977 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.712778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.2.81.66:33113)
0924 12:46:23.713345 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.713352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.713378 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:46:23.713443 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.713450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.713649 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.713787 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.715006 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.716001 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.717340 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.717893 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.718010 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.722297 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.722311 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.722327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.722369 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.725315 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.726194 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.726203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.726205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.726339 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.726643 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.726646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.726648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.726989 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.727201 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.727466 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.727668 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":287,"threads_started":1}
I20260924 12:46:23.729298 6302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:23.811626 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6179
I20260924 12:46:23.823849 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6312
Sep 24 12:46:23 dist-test-slave-9tm1 krb5kdc[5939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:46:23.840940 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:23.824843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.2.81.67:45085)
0924 12:46:23.825508 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:23.825515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:23.825542 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:46:23.825604 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:23.825610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:23.825854 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:23.825983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:23.827295 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.828513 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.829995 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:23.830606 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:23.830716 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:23.835342 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:23.835357 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:23.835372 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:23.835418 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:23.838158 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.838867 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.838874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.838876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.838999 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:23.839478 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:23.839483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:23.839485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:23.839860 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:23.840100 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:23.840361 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 12:46:23.840562 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":369,"threads_started":1}
I20260924 12:46:23.842314 6435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:23.931685 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6312
I20260924 12:46:23.944012 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 5952
I20260924 12:46:23.945289 6014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:24.048566 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 5952
2026-09-24T12:46:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3812 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:24 dist-test-slave-9tm1 krb5kdc[6459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:24 dist-test-slave-9tm1 krb5kdc[6459](info): set up 2 sockets
Sep 24 12:46:24 dist-test-slave-9tm1 krb5kdc[6459](info): commencing operation
krb5kdc: starting...
W20260924 12:46:25.069634 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:25 dist-test-slave-9tm1 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:25.274452 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:42153
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:25.607991 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:25.608323 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:25.608467 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:25.616217 6472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:25.616340 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:25.616390 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:25.616468 6472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:25.616531 6472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:25.626772 6472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:42153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:25.628082 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:25.629415 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:25.641443 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:25.641664 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:25.641812 6480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:25.642452 6472 server_base.cc:1034] running on GCE node
I20260924 12:46:25.643312 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:25.645054 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:25.646311 6472 hybrid_clock.cc:648] HybridClock initialized: now 1790253985646226 us; error 89 us; skew 500 ppm
I20260924 12:46:25.646735 6472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:25 dist-test-slave-9tm1 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253985, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:25.650018 6472 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:25.651363 6472 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:25.651496 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:25.651839 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:25.655027 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "61fd43463865485784e2ccf0267cd89c"
format_stamp: "Formatted at 2026-09-24 12:46:25 on dist-test-slave-9tm1"
server_key: "08118ee8e27eba8e77161a2835cfc437"
server_key_iv: "f23d42d270b3efe5e8fb49ca4bb11d31"
server_key_version: "encryptionkey@0"
I20260924 12:46:25.656090 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "61fd43463865485784e2ccf0267cd89c"
format_stamp: "Formatted at 2026-09-24 12:46:25 on dist-test-slave-9tm1"
server_key: "08118ee8e27eba8e77161a2835cfc437"
server_key_iv: "f23d42d270b3efe5e8fb49ca4bb11d31"
server_key_version: "encryptionkey@0"
I20260924 12:46:25.661818 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:46:25.666054 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:25.667676 6472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:25.667902 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "61fd43463865485784e2ccf0267cd89c"
format_stamp: "Formatted at 2026-09-24 12:46:25 on dist-test-slave-9tm1"
server_key: "08118ee8e27eba8e77161a2835cfc437"
server_key_iv: "f23d42d270b3efe5e8fb49ca4bb11d31"
server_key_version: "encryptionkey@0"
I20260924 12:46:25.668139 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:25.697165 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:25.702422 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:25.702828 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:25.721495 6472 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:42153
I20260924 12:46:25.721477 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:42153 every 8 connection(s)
I20260924 12:46:25.722412 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:25.724960 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6472
I20260924 12:46:25.725237 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:25.725837 2373 external_mini_cluster.cc:1428] Setting key 223ba4c2c85490a45d3c30021fe5ee1d
I20260924 12:46:25.726289 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:25.740613 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c: Bootstrap starting.
Sep 24 12:46:25 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:25.744127 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:25.745438 6538 log.cc:824] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:25.749128 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c: No bootstrap required, opened a new log
I20260924 12:46:25.751703 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:25.727846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.2.81.126:42153)
0924 12:46:25.728677 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:25.728689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:25.728729 (+ 40us) server_negotiation.cc:407] Connection header received
0924 12:46:25.731260 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:25.731289 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:25.731779 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:25.732188 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:25.734148 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:25.735534 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:25.737397 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:25.738257 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:25.738403 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:25.742866 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:25.742876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:25.742892 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:25.742938 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:25.746744 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:25.747918 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:25.747925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:25.747932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:25.748125 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:25.748636 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:25.748641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:25.748643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:25.749355 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:25.749602 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:25.750150 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 12:46:25.750592 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":396,"threads_started":1}
I20260924 12:46:25.757652 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fd43463865485784e2ccf0267cd89c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42153 } }
I20260924 12:46:25.758117 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:25.758214 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61fd43463865485784e2ccf0267cd89c, State: Initialized, Role: FOLLOWER
I20260924 12:46:25.758984 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fd43463865485784e2ccf0267cd89c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42153 } }
I20260924 12:46:25.759246 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:25.759361 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:25.759500 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:25.762295 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fd43463865485784e2ccf0267cd89c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42153 } }
I20260924 12:46:25.762837 6538 leader_election.cc:302] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61fd43463865485784e2ccf0267cd89c; no voters:
I20260924 12:46:25.763661 6538 leader_election.cc:288] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:25.763787 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:25.764550 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [term 1 LEADER]: Becoming Leader. State: Replica: 61fd43463865485784e2ccf0267cd89c, State: Running, Role: LEADER
I20260924 12:46:25.765352 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fd43463865485784e2ccf0267cd89c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42153 } }
I20260924 12:46:25.766179 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:25.768118 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61fd43463865485784e2ccf0267cd89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fd43463865485784e2ccf0267cd89c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42153 } } }
I20260924 12:46:25.768522 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:25.768065 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61fd43463865485784e2ccf0267cd89c. Latest consensus state: current_term: 1 leader_uuid: "61fd43463865485784e2ccf0267cd89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fd43463865485784e2ccf0267cd89c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42153 } } }
I20260924 12:46:25.769176 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:25.769529 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:25.781196 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:25.791821 6550 catalog_manager.cc:1347] Generated new cluster ID: 566faf50f8164e899895df3702efec39
I20260924 12:46:25.791950 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:25.806716 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:25.808013 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:25.815980 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c: Generated new TSK 0
I20260924 12:46:25.816804 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:46:25.869088 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42153
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:26.209692 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:26.209987 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:26.210111 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:26.217469 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:26.217594 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:26.217752 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:26.227573 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:26.228840 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:26.230136 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:26.242766 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:26.242796 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:26.242724 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:26.243427 6564 server_base.cc:1034] running on GCE node
I20260924 12:46:26.244019 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:26.244966 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:26.246253 6564 hybrid_clock.cc:648] HybridClock initialized: now 1790253986246176 us; error 46 us; skew 500 ppm
I20260924 12:46:26.246663 6564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:26 dist-test-slave-9tm1 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:26.249926 6564 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:46:26.251294 6564 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:26.251394 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:26.251691 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:26.254724 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "3365e6a980304d759bc24d9e6c2f4006"
format_stamp: "Formatted at 2026-09-24 12:46:26 on dist-test-slave-9tm1"
server_key: "004df30703fd3fc088f2d9b62aeb09e7"
server_key_iv: "b13615727efe43fc75410190f7dd256a"
server_key_version: "encryptionkey@0"
I20260924 12:46:26.255568 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "3365e6a980304d759bc24d9e6c2f4006"
format_stamp: "Formatted at 2026-09-24 12:46:26 on dist-test-slave-9tm1"
server_key: "004df30703fd3fc088f2d9b62aeb09e7"
server_key_iv: "b13615727efe43fc75410190f7dd256a"
server_key_version: "encryptionkey@0"
I20260924 12:46:26.261312 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:46:26.265587 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:26.267386 6564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:46:26.267568 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "3365e6a980304d759bc24d9e6c2f4006"
format_stamp: "Formatted at 2026-09-24 12:46:26 on dist-test-slave-9tm1"
server_key: "004df30703fd3fc088f2d9b62aeb09e7"
server_key_iv: "b13615727efe43fc75410190f7dd256a"
server_key_version: "encryptionkey@0"
I20260924 12:46:26.267810 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:26.304289 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:26.308614 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:26.308882 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:26.309948 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:26.311276 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:26.311373 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:26.311501 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:26.311573 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:26.342276 6564 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:44191
I20260924 12:46:26.342348 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:44191 every 8 connection(s)
I20260924 12:46:26.343118 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:26.350847 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6564
I20260924 12:46:26.351039 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:26.351686 2373 external_mini_cluster.cc:1428] Setting key 2a67d92d29d715eaa2d8f39c00c123cd
Sep 24 12:46:26 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:26.364347 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:26.345529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:58855 (local address 127.2.81.126:42153)
0924 12:46:26.346034 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:26.346040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:26.346914 (+ 874us) server_negotiation.cc:407] Connection header received
0924 12:46:26.348799 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:26.348804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:26.348901 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:26.349029 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:26.350676 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.351460 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.353504 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.353969 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.354106 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:26.358177 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:26.358185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:26.358189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:26.358232 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:26.360943 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:26.361935 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:26.361941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:26.361943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:26.362105 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:26.362576 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:26.362581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:26.362583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:26.362881 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:26.363003 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:26.363911 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 12:46:26.364102 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":301,"threads_started":1}
I20260924 12:46:26.365669 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:26.346004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:42153 (local address 127.2.81.65:58855)
0924 12:46:26.346729 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:26.346769 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:26.348459 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:26.349295 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:26.349304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:26.349858 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:26.350466 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:26.350487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.351656 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.351660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:26.353366 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:26.353382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.354284 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.354290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:26.354511 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:26.355332 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:26.355358 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:26.357877 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:26.361159 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:26.361170 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:26.361192 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:26.361806 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:26.362210 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:26.362215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:26.362219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:26.362444 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:26.363006 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:26.363014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:26.363453 (+ 439us) client_negotiation.cc:769] Sending connection context
0924 12:46:26.363773 (+ 320us) client_negotiation.cc:240] Negotiation successful
0924 12:46:26.364314 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":294,"threads_started":1}
I20260924 12:46:26.367265 6691 heartbeater.cc:347] Connected to a master server at 127.2.81.126:42153
I20260924 12:46:26.367609 6691 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:26.368429 6691 heartbeater.cc:511] Master 127.2.81.126:42153 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:46:26.370771 6503 ts_manager.cc:194] Registered new tserver with Master: 3365e6a980304d759bc24d9e6c2f4006 (127.2.81.65:44191)
I20260924 12:46:26.372493 6503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:58855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:26.396824 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42153
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:26.735868 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:26.736172 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:26.736435 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:26.744779 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:26.745003 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:26.745184 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:26.755581 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:26.757114 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:26.758530 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:26.773316 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:26.773831 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:26.773878 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:26.774714 6697 server_base.cc:1034] running on GCE node
I20260924 12:46:26.775599 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:26.776938 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:26.778216 6697 hybrid_clock.cc:648] HybridClock initialized: now 1790253986778143 us; error 62 us; skew 500 ppm
I20260924 12:46:26.778626 6697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:26 dist-test-slave-9tm1 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:26.786593 6697 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:26.788246 6697 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:26.788393 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:26.788777 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:26.793182 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "e24617aebb50439eaf5a87ea931f582e"
format_stamp: "Formatted at 2026-09-24 12:46:26 on dist-test-slave-9tm1"
server_key: "086420c91a91e2df982d070855671878"
server_key_iv: "87fbde1e8de9f5eb6889de94b1fed2de"
server_key_version: "encryptionkey@0"
I20260924 12:46:26.794440 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "e24617aebb50439eaf5a87ea931f582e"
format_stamp: "Formatted at 2026-09-24 12:46:26 on dist-test-slave-9tm1"
server_key: "086420c91a91e2df982d070855671878"
server_key_iv: "87fbde1e8de9f5eb6889de94b1fed2de"
server_key_version: "encryptionkey@0"
I20260924 12:46:26.802116 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260924 12:46:26.807468 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:26.809427 6697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 12:46:26.809652 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "e24617aebb50439eaf5a87ea931f582e"
format_stamp: "Formatted at 2026-09-24 12:46:26 on dist-test-slave-9tm1"
server_key: "086420c91a91e2df982d070855671878"
server_key_iv: "87fbde1e8de9f5eb6889de94b1fed2de"
server_key_version: "encryptionkey@0"
I20260924 12:46:26.809955 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:26.843707 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:26.850044 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:26.850412 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:26.851794 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:26.853749 6697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:26.853876 6697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:26.853988 6697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:26.854066 6697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:26.889430 6697 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:40443
I20260924 12:46:26.889484 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:40443 every 8 connection(s)
I20260924 12:46:26.890574 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:26.899763 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6697
I20260924 12:46:26.899963 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:26.900462 2373 external_mini_cluster.cc:1428] Setting key 224e0ae330bbc8f5b2072d227f4d3252
Sep 24 12:46:26 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:26.913578 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:26.893130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:55817 (local address 127.2.81.126:42153)
0924 12:46:26.893601 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:26.893608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:26.894582 (+ 974us) server_negotiation.cc:407] Connection header received
0924 12:46:26.896626 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:26.896633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:26.896723 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:26.896848 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:26.898496 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.899602 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.901762 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.902232 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.902340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:26.906678 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:26.906684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:26.906689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:26.906728 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:26.909809 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:26.910857 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:26.910863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:26.910866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:26.911017 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:26.911549 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:26.911554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:26.911558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:26.911880 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:26.912016 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:26.912968 (+ 952us) server_negotiation.cc:299] Negotiation successful
0924 12:46:26.913242 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260924 12:46:26.914983 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:26.893648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:42153 (local address 127.2.81.66:55817)
0924 12:46:26.894410 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:26.894460 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:26.896292 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:26.897145 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:26.897153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:26.897722 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:26.898306 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:26.898325 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.899782 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.899788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:26.901620 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:26.901632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:26.902443 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:26.902451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:26.902656 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:26.903449 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:26.903479 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:26.906374 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:26.909995 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:26.910008 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:26.910028 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:26.910704 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:26.911135 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:26.911141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:26.911145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:26.911421 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:26.912007 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:26.912017 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:26.912517 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 12:46:26.912854 (+ 337us) client_negotiation.cc:240] Negotiation successful
0924 12:46:26.913486 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":317,"threads_started":1}
I20260924 12:46:26.916675 6824 heartbeater.cc:347] Connected to a master server at 127.2.81.126:42153
I20260924 12:46:26.917254 6824 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:26.918247 6824 heartbeater.cc:511] Master 127.2.81.126:42153 requested a full tablet report, sending...
I20260924 12:46:26.920164 6503 ts_manager.cc:194] Registered new tserver with Master: e24617aebb50439eaf5a87ea931f582e (127.2.81.66:40443)
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:46:26.921422 6503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:55817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:26.954273 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42153
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:27.288940 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:27.289324 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:27.289456 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:27.297094 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:27.297226 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:27.297389 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:27.307514 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:27.308898 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:27.310312 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:27.323223 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:27.323242 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:27.323274 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:27.323832 6830 server_base.cc:1034] running on GCE node
I20260924 12:46:27.324508 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:27.325493 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:27.326754 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790253987326696 us; error 52 us; skew 500 ppm
I20260924 12:46:27.327102 6830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:27 dist-test-slave-9tm1 krb5kdc[6459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253987, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:27.330152 6830 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:27.331279 6830 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:27.331384 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:27.331727 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:27.334648 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "d5d2c5cb083d4b6fb497a3185f312eaa"
format_stamp: "Formatted at 2026-09-24 12:46:27 on dist-test-slave-9tm1"
server_key: "2ae30b0cdfafe5af8f342a014631be1b"
server_key_iv: "bf2dc14df26b682cf46da32c7c28bc64"
server_key_version: "encryptionkey@0"
I20260924 12:46:27.335443 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "d5d2c5cb083d4b6fb497a3185f312eaa"
format_stamp: "Formatted at 2026-09-24 12:46:27 on dist-test-slave-9tm1"
server_key: "2ae30b0cdfafe5af8f342a014631be1b"
server_key_iv: "bf2dc14df26b682cf46da32c7c28bc64"
server_key_version: "encryptionkey@0"
I20260924 12:46:27.340636 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:46:27.344450 6844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:27.345919 6830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:27.346077 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "d5d2c5cb083d4b6fb497a3185f312eaa"
format_stamp: "Formatted at 2026-09-24 12:46:27 on dist-test-slave-9tm1"
server_key: "2ae30b0cdfafe5af8f342a014631be1b"
server_key_iv: "bf2dc14df26b682cf46da32c7c28bc64"
server_key_version: "encryptionkey@0"
I20260924 12:46:27.346284 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:27.363704 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:27.368069 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:27.368321 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:27.369314 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:27.370656 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:27.370754 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:27.370844 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:27.370910 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:27.376215 6691 heartbeater.cc:503] Master 127.2.81.126:42153 was elected leader, sending a full tablet report...
I20260924 12:46:27.403338 6830 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:44247
I20260924 12:46:27.403381 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:44247 every 8 connection(s)
I20260924 12:46:27.404135 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:27.413384 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 6830
I20260924 12:46:27.413599 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:27.413991 2373 external_mini_cluster.cc:1428] Setting key 00c92126f585cf85a51e002b6c1b9431
Sep 24 12:46:27 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253987, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:27.424261 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.406466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:59135 (local address 127.2.81.126:42153)
0924 12:46:27.406658 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.406664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.407807 (+ 1143us) server_negotiation.cc:407] Connection header received
0924 12:46:27.409648 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.409652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.409740 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.409834 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:27.411344 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.412161 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.414296 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.414831 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.414967 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.418798 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:27.418804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:27.418807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:27.418840 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:27.421339 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:27.422071 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:27.422075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:27.422077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:27.422212 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:27.422591 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:27.422595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:27.422597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:27.422934 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:27.423056 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.423835 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.424017 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 12:46:27.425300 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:42153 (local address 127.2.81.67:59135)
0924 12:46:27.407648 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:27.407688 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:27.409330 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:27.410016 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:27.410024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:27.410580 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:27.411135 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.411157 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.412332 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.412338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.414122 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.414138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.415064 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.415072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.415403 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.416100 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:27.416123 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:27.418593 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:27.421471 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:27.421484 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:27.421501 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:27.421946 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:27.422294 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:27.422297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:27.422299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:27.422466 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:27.423072 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:27.423079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:27.423410 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 12:46:27.423685 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 12:46:27.424121 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":319,"threads_started":1}
I20260924 12:46:27.426919 6957 heartbeater.cc:347] Connected to a master server at 127.2.81.126:42153
I20260924 12:46:27.427289 6957 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:27.427919 6957 heartbeater.cc:511] Master 127.2.81.126:42153 requested a full tablet report, sending...
I20260924 12:46:27.429649 6503 ts_manager.cc:194] Registered new tserver with Master: d5d2c5cb083d4b6fb497a3185f312eaa (127.2.81.67:44247)
I20260924 12:46:27.430365 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:46:27.430576 6503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:59135
I20260924 12:46:27.448040 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.435494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.2.81.126:42153)
0924 12:46:27.435660 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.435665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.436145 (+ 480us) server_negotiation.cc:407] Connection header received
0924 12:46:27.436285 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.436289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.436357 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.436447 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:27.438005 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.438956 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.440222 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.440650 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.440760 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.443045 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:27.443051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:27.443054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:27.443089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:27.445534 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:27.446199 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:27.446204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:27.446208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:27.446328 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:27.446667 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:27.446672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:27.446675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:27.446990 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:27.447232 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.447523 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.447729 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 12:46:27.452195 6503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:27.458938 6503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:27.493717 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.471271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:40443 (local address 127.0.0.1:60460)
0924 12:46:27.481578 (+ 10307us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:27.481600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:27.481746 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:27.484800 (+ 3054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:27.484807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:27.484840 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:27.485303 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.485315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.488165 (+ 2850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.488172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.490591 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.490609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.492682 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.492688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.492905 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.492959 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 12:46:27.493111 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 12:46:27.493336 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10161,"thread_start_us":8558,"threads_started":1}
I20260924 12:46:27.494503 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.471471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.2.81.66:40443)
0924 12:46:27.482959 (+ 11488us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.482967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.482996 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:46:27.483083 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.483090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.483321 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.484802 (+ 1481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:27.485521 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.487998 (+ 2477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.490851 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.492452 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.493479 (+ 1027us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.493674 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.493771 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.494201 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11338,"thread_start_us":11241,"threads_started":1}
I20260924 12:46:27.494550 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.470690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.2.81.67:44247)
0924 12:46:27.482276 (+ 11586us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.482285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.482317 (+ 32us) server_negotiation.cc:407] Connection header received
0924 12:46:27.482411 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.482420 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.482707 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.482920 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:27.484332 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.486076 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.490001 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.493046 (+ 3045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.493153 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.493382 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.494079 (+ 697us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.494261 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11433,"thread_start_us":11292,"threads_started":1}
I20260924 12:46:27.494995 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.470528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:44247 (local address 127.0.0.1:40760)
0924 12:46:27.480328 (+ 9800us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:27.480393 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:27.480621 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:27.483248 (+ 2627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:27.483255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:27.483286 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:27.483708 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.484102 (+ 394us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.486262 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.486268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.489788 (+ 3520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.489807 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.493542 (+ 3735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.493548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.493727 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.493757 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 12:46:27.493834 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 12:46:27.494676 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9471,"thread_start_us":10763,"threads_started":1}
I20260924 12:46:27.498102 6759 tablet_service.cc:1467] Processing CreateTablet for tablet d160f992388c4e40839ac8b6aa08408b (DEFAULT_TABLE table=test-table [id=670c7cd61e374620be362c64efd542c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:27.498458 6892 tablet_service.cc:1467] Processing CreateTablet for tablet d160f992388c4e40839ac8b6aa08408b (DEFAULT_TABLE table=test-table [id=670c7cd61e374620be362c64efd542c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:27.499670 6759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d160f992388c4e40839ac8b6aa08408b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:27.499679 6892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d160f992388c4e40839ac8b6aa08408b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:27.500586 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.482367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35676)
0924 12:46:27.493446 (+ 11079us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:27.493475 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:27.493602 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:27.494312 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:27.494316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:27.494339 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:27.495011 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.495026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.496604 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.496609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.498090 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.498102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.499863 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.499872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.500017 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.500040 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 12:46:27.500121 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 12:46:27.500258 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10963,"thread_start_us":10805,"threads_started":1}
I20260924 12:46:27.500638 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.482037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.2.81.65:44191)
0924 12:46:27.492810 (+ 10773us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.492820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.493724 (+ 904us) server_negotiation.cc:407] Connection header received
0924 12:46:27.493869 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.493879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.494135 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.494318 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:27.495214 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.496468 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.498244 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.499581 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.499701 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.499905 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.500179 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.500337 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10602,"thread_start_us":10785,"threads_started":1}
I20260924 12:46:27.503666 6626 tablet_service.cc:1467] Processing CreateTablet for tablet d160f992388c4e40839ac8b6aa08408b (DEFAULT_TABLE table=test-table [id=670c7cd61e374620be362c64efd542c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:27.505129 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d160f992388c4e40839ac8b6aa08408b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:27.513995 6976 tablet_bootstrap.cc:492] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Bootstrap starting.
I20260924 12:46:27.514781 6975 tablet_bootstrap.cc:492] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Bootstrap starting.
I20260924 12:46:27.516392 6977 tablet_bootstrap.cc:492] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Bootstrap starting.
I20260924 12:46:27.518536 6975 tablet_bootstrap.cc:654] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:27.518641 6976 tablet_bootstrap.cc:654] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:27.519541 6977 tablet_bootstrap.cc:654] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:27.519532 6975 log.cc:824] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:27.519799 6976 log.cc:824] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:27.520665 6977 log.cc:824] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:27.526064 6975 tablet_bootstrap.cc:492] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: No bootstrap required, opened a new log
I20260924 12:46:27.526146 6976 tablet_bootstrap.cc:492] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: No bootstrap required, opened a new log
I20260924 12:46:27.526196 6977 tablet_bootstrap.cc:492] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: No bootstrap required, opened a new log
I20260924 12:46:27.526361 6975 ts_tablet_manager.cc:1397] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 12:46:27.526536 6976 ts_tablet_manager.cc:1397] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260924 12:46:27.526523 6977 ts_tablet_manager.cc:1397] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 12:46:27.534161 6975 raft_consensus.cc:348] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.534549 6975 raft_consensus.cc:374] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:27.534636 6975 raft_consensus.cc:729] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24617aebb50439eaf5a87ea931f582e, State: Initialized, Role: FOLLOWER
I20260924 12:46:27.535243 6975 consensus_queue.cc:260] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.537084 6976 raft_consensus.cc:348] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.537083 6977 raft_consensus.cc:348] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.537556 6977 raft_consensus.cc:374] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:27.537557 6976 raft_consensus.cc:374] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:27.537665 6977 raft_consensus.cc:729] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3365e6a980304d759bc24d9e6c2f4006, State: Initialized, Role: FOLLOWER
I20260924 12:46:27.537665 6976 raft_consensus.cc:729] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d2c5cb083d4b6fb497a3185f312eaa, State: Initialized, Role: FOLLOWER
I20260924 12:46:27.538022 6824 heartbeater.cc:503] Master 127.2.81.126:42153 was elected leader, sending a full tablet report...
I20260924 12:46:27.538450 6975 ts_tablet_manager.cc:1428] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 12:46:27.538321 6976 consensus_queue.cc:260] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.539395 6957 heartbeater.cc:503] Master 127.2.81.126:42153 was elected leader, sending a full tablet report...
I20260924 12:46:27.540254 6976 ts_tablet_manager.cc:1428] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 12:46:27.540059 6977 consensus_queue.cc:260] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.542064 6977 ts_tablet_manager.cc:1428] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260924 12:46:27.595378 6692 tablet.cc:2369] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:27.642594 6825 tablet.cc:2369] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:27.655543 6958 tablet.cc:2369] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:27.774439 6982 raft_consensus.cc:482] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:27.774765 6982 raft_consensus.cc:504] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.776517 6982 leader_election.cc:288] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3365e6a980304d759bc24d9e6c2f4006 (127.2.81.65:44191), e24617aebb50439eaf5a87ea931f582e (127.2.81.66:40443)
I20260924 12:46:27.782684 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.776914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:55631 (local address 127.2.81.65:44191)
0924 12:46:27.777127 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.777134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.777155 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:46:27.777358 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.777364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.777439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.777525 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:27.777888 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.778806 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.780626 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.781824 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.781929 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.782014 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.782366 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.782471 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 12:46:27.782747 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.776699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.2.81.67:55631)
0924 12:46:27.777039 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:27.777061 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:27.777182 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:27.777578 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:27.777583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:27.777613 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:27.777772 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.777780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.778947 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.778952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.780458 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.780469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.781976 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.781982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.782168 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.782204 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:46:27.782304 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 12:46:27.782444 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":2}
I20260924 12:46:27.784055 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.777120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:34395 (local address 127.2.81.66:40443)
0924 12:46:27.777343 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.777350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.777781 (+ 431us) server_negotiation.cc:407] Connection header received
0924 12:46:27.777949 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.777953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.778007 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.778070 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:27.778572 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.779481 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.781903 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.783063 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.783189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.783280 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.783676 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.783815 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:46:27.784138 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.776921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:40443 (local address 127.2.81.67:34395)
0924 12:46:27.777686 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:27.777703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:27.777809 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:27.778105 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:27.778109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:27.778133 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:27.778423 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.778434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.779729 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.779734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.781711 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:27.781727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.783272 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.783279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:27.783437 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.783469 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 12:46:27.783598 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 12:46:27.783709 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":38,"thread_start_us":428,"threads_started":1}
I20260924 12:46:27.784142 6646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d160f992388c4e40839ac8b6aa08408b" candidate_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3365e6a980304d759bc24d9e6c2f4006" is_pre_election: true
I20260924 12:46:27.784871 6646 raft_consensus.cc:2454] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5d2c5cb083d4b6fb497a3185f312eaa in term 0.
I20260924 12:46:27.785368 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d160f992388c4e40839ac8b6aa08408b" candidate_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24617aebb50439eaf5a87ea931f582e" is_pre_election: true
I20260924 12:46:27.785619 6847 leader_election.cc:302] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3365e6a980304d759bc24d9e6c2f4006, d5d2c5cb083d4b6fb497a3185f312eaa; no voters:
I20260924 12:46:27.786024 6982 raft_consensus.cc:2781] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:27.786001 6779 raft_consensus.cc:2454] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5d2c5cb083d4b6fb497a3185f312eaa in term 0.
I20260924 12:46:27.786176 6982 raft_consensus.cc:482] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:27.786291 6982 raft_consensus.cc:3037] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:27.788707 6982 raft_consensus.cc:504] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.789352 6982 leader_election.cc:288] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [CANDIDATE]: Term 1 election: Requested vote from peers 3365e6a980304d759bc24d9e6c2f4006 (127.2.81.65:44191), e24617aebb50439eaf5a87ea931f582e (127.2.81.66:40443)
I20260924 12:46:27.789975 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d160f992388c4e40839ac8b6aa08408b" candidate_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24617aebb50439eaf5a87ea931f582e"
I20260924 12:46:27.789979 6646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d160f992388c4e40839ac8b6aa08408b" candidate_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3365e6a980304d759bc24d9e6c2f4006"
I20260924 12:46:27.790179 6779 raft_consensus.cc:3037] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:27.790192 6646 raft_consensus.cc:3037] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:27.792614 6779 raft_consensus.cc:2454] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d2c5cb083d4b6fb497a3185f312eaa in term 1.
I20260924 12:46:27.792614 6646 raft_consensus.cc:2454] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d2c5cb083d4b6fb497a3185f312eaa in term 1.
I20260924 12:46:27.793149 6847 leader_election.cc:302] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3365e6a980304d759bc24d9e6c2f4006, d5d2c5cb083d4b6fb497a3185f312eaa; no voters:
I20260924 12:46:27.793478 6982 raft_consensus.cc:2781] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:27.793962 6982 raft_consensus.cc:686] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 1 LEADER]: Becoming Leader. State: Replica: d5d2c5cb083d4b6fb497a3185f312eaa, State: Running, Role: LEADER
I20260924 12:46:27.794448 6982 consensus_queue.cc:237] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } }
I20260924 12:46:27.799567 6503 catalog_manager.cc:5495] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to d5d2c5cb083d4b6fb497a3185f312eaa (127.2.81.67). New cstate: current_term: 1 leader_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d2c5cb083d4b6fb497a3185f312eaa" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 44247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:27.858446 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.853502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.2.81.67:44247)
0924 12:46:27.853680 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.853686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.853704 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:27.853793 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.853796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.853866 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.853998 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:27.854365 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.855067 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.856372 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.856764 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.856873 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.857383 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:27.857895 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:27.858022 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.858113 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.858195 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 12:46:27.866329 6646 raft_consensus.cc:1261] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 1 FOLLOWER]: Refusing update from remote peer d5d2c5cb083d4b6fb497a3185f312eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:27.866329 6779 raft_consensus.cc:1261] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 1 FOLLOWER]: Refusing update from remote peer d5d2c5cb083d4b6fb497a3185f312eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:27.867323 6985 consensus_queue.cc:1035] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24617aebb50439eaf5a87ea931f582e" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:27.867723 6982 consensus_queue.cc:1035] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3365e6a980304d759bc24d9e6c2f4006" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 44191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:27.885453 6988 mvcc.cc:205] Tried to move back new op lower bound from 7332880334290411520 to 7332880334009831424. Current Snapshot: MvccSnapshot[applied={T|T < 7332880334290411520}]
I20260924 12:46:27.899277 6503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:27.899528 6503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:27.902798 6503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61fd43463865485784e2ccf0267cd89c: Sending DeleteTablet for 3 replicas of tablet d160f992388c4e40839ac8b6aa08408b
I20260924 12:46:27.904026 6626 tablet_service.cc:1514] Processing DeleteTablet for tablet d160f992388c4e40839ac8b6aa08408b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:27 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:35676
I20260924 12:46:27.904233 6892 tablet_service.cc:1514] Processing DeleteTablet for tablet d160f992388c4e40839ac8b6aa08408b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:27 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:40760
I20260924 12:46:27.905551 6759 tablet_service.cc:1514] Processing DeleteTablet for tablet d160f992388c4e40839ac8b6aa08408b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:27 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:60460
I20260924 12:46:27.909283 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6564
I20260924 12:46:27.915843 6999 tablet_replica.cc:330] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: stopping tablet replica
I20260924 12:46:27.916945 7000 tablet_replica.cc:330] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: stopping tablet replica
I20260924 12:46:27.917279 6999 raft_consensus.cc:2229] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:27.917367 7000 raft_consensus.cc:2229] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:27.917900 7000 raft_consensus.cc:2258] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:27.917907 6999 raft_consensus.cc:2258] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:27.920017 6999 ts_tablet_manager.cc:1905] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:27.920017 7000 ts_tablet_manager.cc:1905] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:27.926107 6998 tablet_replica.cc:330] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: stopping tablet replica
I20260924 12:46:27.927698 7000 ts_tablet_manager.cc:1918] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:27.927901 7000 log.cc:1196] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/d160f992388c4e40839ac8b6aa08408b
I20260924 12:46:27.928438 7000 ts_tablet_manager.cc:1939] T d160f992388c4e40839ac8b6aa08408b P e24617aebb50439eaf5a87ea931f582e: Deleting consensus metadata
I20260924 12:46:27.928552 6998 raft_consensus.cc:2229] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 24 12:46:27 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:27.929102 6998 raft_consensus.cc:2258] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:27.930402 6490 catalog_manager.cc:4841] TS e24617aebb50439eaf5a87ea931f582e (127.2.81.66:40443): tablet d160f992388c4e40839ac8b6aa08408b (table test-table [id=670c7cd61e374620be362c64efd542c1]) successfully deleted
I20260924 12:46:27.934787 6998 ts_tablet_manager.cc:1905] T d160f992388c4e40839ac8b6aa08408b P 3365e6a980304d759bc24d9e6c2f4006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:27.937089 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:27.910409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.2.81.65:44191)
0924 12:46:27.910580 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:27.910586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:27.917562 (+ 6976us) server_negotiation.cc:407] Connection header received
0924 12:46:27.917889 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:27.917896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:27.918001 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:27.918092 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:27.920305 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.921625 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.923674 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:27.924666 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:27.924777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:27.930317 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:27.930331 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:27.930349 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:27.930399 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:27.933403 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:27.934266 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:27.934276 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:27.934279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:27.934457 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:27.935015 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:27.935020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:27.935023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:27.935488 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:27.935780 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:27.936407 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 12:46:27.936659 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:46:27.937285 6999 ts_tablet_manager.cc:1918] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:27.937539 6999 log.cc:1196] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/d160f992388c4e40839ac8b6aa08408b
I20260924 12:46:27.938114 6999 ts_tablet_manager.cc:1939] T d160f992388c4e40839ac8b6aa08408b P d5d2c5cb083d4b6fb497a3185f312eaa: Deleting consensus metadata
I20260924 12:46:27.938871 6686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:27.940001 6489 catalog_manager.cc:4841] TS d5d2c5cb083d4b6fb497a3185f312eaa (127.2.81.67:44247): tablet d160f992388c4e40839ac8b6aa08408b (table test-table [id=670c7cd61e374620be362c64efd542c1]) successfully deleted
I20260924 12:46:28.029202 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6564
I20260924 12:46:28.043107 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6697
W20260924 12:46:28.043179 6489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:44191 (error 108)
I20260924 12:46:28.045645 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.044694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35700)
0924 12:46:28.045248 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.045370 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":332,"threads_started":1}
W20260924 12:46:28.045990 6489 catalog_manager.cc:4584] TS 3365e6a980304d759bc24d9e6c2f4006 (127.2.81.65:44191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d160f992388c4e40839ac8b6aa08408b: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Sep 24 12:46:28 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:46:28.058842 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.044186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.2.81.66:40443)
0924 12:46:28.044357 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:28.044363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:28.044436 (+ 73us) server_negotiation.cc:407] Connection header received
0924 12:46:28.044630 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:28.044634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:28.044706 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:28.044788 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:28.046280 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:28.046955 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:28.048143 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:28.048532 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:28.048649 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:28.053204 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:28.053219 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:28.053235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:28.053275 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:28.056036 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:28.056750 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:28.056758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:28.056761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:28.056898 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:28.057296 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:28.057300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:28.057302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:28.057693 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:28.057961 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:28.058296 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 12:46:28.058501 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:28.059474 6819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:28.099668 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.099217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35704)
0924 12:46:28.099426 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.099518 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 12:46:28.100803 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.100425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35714)
0924 12:46:28.100592 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.100664 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 12:46:28.161536 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.161052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35730)
0924 12:46:28.161287 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.161386 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 12:46:28.162715 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.162292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35732)
0924 12:46:28.162461 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.162561 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 12:46:28.225252 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6697
I20260924 12:46:28.240437 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6830
I20260924 12:46:28.243788 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.243030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35740)
0924 12:46:28.243271 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.243388 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 12:46:28.245306 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.244643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:44191 (local address 127.0.0.1:35746)
0924 12:46:28.245019 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:28.245128 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
Sep 24 12:46:28 dist-test-slave-9tm1 krb5kdc[6459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:46:28.261771 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.241256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.2.81.67:44247)
0924 12:46:28.241460 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:28.241467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:28.241567 (+ 100us) server_negotiation.cc:407] Connection header received
0924 12:46:28.241729 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:28.241735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:28.241831 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:28.241969 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:28.243704 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:28.245540 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:28.247205 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:28.247721 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:28.247960 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:28.253391 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:28.253406 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:28.253423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:28.253476 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:28.258025 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:28.258882 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:28.258891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:28.258893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:28.259198 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:28.259823 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:28.259831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:28.259835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:28.260277 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:28.260583 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:28.261073 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 12:46:28.261274 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:28.263022 6952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:28.387166 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6830
I20260924 12:46:28.401731 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 6472
I20260924 12:46:28.402840 6533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:28.508431 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 6472
2026-09-24T12:46:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4458 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T12:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:28Z Disabled control of system clock
I20260924 12:46:28.568902 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33103
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 12:46:28.887692 7011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:28.887995 7011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:28.888087 7011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:28.895385 7011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:28.895498 7011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:28.895542 7011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:28.895602 7011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:28.895645 7011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:28.904820 7011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:33103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:28.906335 7011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:28.908232 7011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:28.919662 7017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:28.919783 7019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:28.919641 7016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:28.920478 7011 server_base.cc:1034] running on GCE node
I20260924 12:46:28.921295 7011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:28.922777 7011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:28.924009 7011 hybrid_clock.cc:648] HybridClock initialized: now 1790253988923957 us; error 49 us; skew 500 ppm
I20260924 12:46:28.924381 7011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:28.926839 7011 webserver.cc:460] Webserver started at http://127.2.81.126:43287/ using document root <none> and password file <none>
I20260924 12:46:28.927549 7011 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:28.927654 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:28.927959 7011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:28.931051 7011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "f5767121ad344ec9a10e79caddfc6a63"
format_stamp: "Formatted at 2026-09-24 12:46:28 on dist-test-slave-9tm1"
server_key: "1fd0d3d15359766b4b93bbcd848b4b1e"
server_key_iv: "d5a9ca3335b25576fae3ce442a65b68e"
server_key_version: "encryptionkey@0"
I20260924 12:46:28.931890 7011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "f5767121ad344ec9a10e79caddfc6a63"
format_stamp: "Formatted at 2026-09-24 12:46:28 on dist-test-slave-9tm1"
server_key: "1fd0d3d15359766b4b93bbcd848b4b1e"
server_key_iv: "d5a9ca3335b25576fae3ce442a65b68e"
server_key_version: "encryptionkey@0"
I20260924 12:46:28.937541 7011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 12:46:28.941123 7025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:28.942538 7011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:28.942713 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "f5767121ad344ec9a10e79caddfc6a63"
format_stamp: "Formatted at 2026-09-24 12:46:28 on dist-test-slave-9tm1"
server_key: "1fd0d3d15359766b4b93bbcd848b4b1e"
server_key_iv: "d5a9ca3335b25576fae3ce442a65b68e"
server_key_version: "encryptionkey@0"
I20260924 12:46:28.942932 7011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:28.948735 7011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:28.949576 7011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:28.949856 7011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:28.967931 7011 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:33103
I20260924 12:46:28.967953 7076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:33103 every 8 connection(s)
I20260924 12:46:28.969514 7011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:28.973392 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:28.976750 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7011
I20260924 12:46:28.976959 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:28.977751 2373 external_mini_cluster.cc:1428] Setting key 35faf9fb79735c4161b991e7aea16134
I20260924 12:46:28.987146 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:28.980126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.2.81.126:33103)
0924 12:46:28.981409 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:28.981422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:28.981473 (+ 51us) server_negotiation.cc:407] Connection header received
0924 12:46:28.983228 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:28.983253 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:28.983771 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:28.984152 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:28.985352 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:28.985358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:28.985377 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:28.985391 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:28.985460 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:28.985476 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:28.985642 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:28.985784 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 12:46:28.986057 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":369,"threads_started":1}
I20260924 12:46:28.987514 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63: Bootstrap starting.
I20260924 12:46:28.990908 7077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:28.991889 7077 log.cc:824] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:28.994807 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63: No bootstrap required, opened a new log
I20260924 12:46:29.001919 7077 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5767121ad344ec9a10e79caddfc6a63" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33103 } }
I20260924 12:46:29.002362 7077 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:29.002441 7077 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5767121ad344ec9a10e79caddfc6a63, State: Initialized, Role: FOLLOWER
I20260924 12:46:29.002992 7077 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5767121ad344ec9a10e79caddfc6a63" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33103 } }
I20260924 12:46:29.003217 7077 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:29.003327 7077 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:29.003448 7077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:29.005671 7077 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5767121ad344ec9a10e79caddfc6a63" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33103 } }
I20260924 12:46:29.006192 7077 leader_election.cc:302] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5767121ad344ec9a10e79caddfc6a63; no voters:
I20260924 12:46:29.006762 7077 leader_election.cc:288] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:29.006914 7082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:29.007570 7082 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [term 1 LEADER]: Becoming Leader. State: Replica: f5767121ad344ec9a10e79caddfc6a63, State: Running, Role: LEADER
I20260924 12:46:29.008111 7082 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5767121ad344ec9a10e79caddfc6a63" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33103 } }
I20260924 12:46:29.008733 7077 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:29.010851 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5767121ad344ec9a10e79caddfc6a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5767121ad344ec9a10e79caddfc6a63" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33103 } } }
I20260924 12:46:29.011291 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:29.011144 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5767121ad344ec9a10e79caddfc6a63. Latest consensus state: current_term: 1 leader_uuid: "f5767121ad344ec9a10e79caddfc6a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5767121ad344ec9a10e79caddfc6a63" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33103 } } }
I20260924 12:46:29.011891 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:29.012838 7089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:29.017367 7089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:29.027355 7089 catalog_manager.cc:1347] Generated new cluster ID: 51cfc4d93a0047f183ce89a84755530b
I20260924 12:46:29.027541 7089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:29.037127 7089 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:29.037617 7089 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:29.054203 7089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63: Generated new TSK 0
I20260924 12:46:29.055100 7089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:29.063086 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33103
--builtin_ntp_servers=127.2.81.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 12:46:29.394536 7101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:29.394845 7101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:29.394955 7101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:29.402845 7101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:29.402973 7101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:29.403116 7101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:29.412644 7101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:29.414218 7101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:29.416129 7101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:29.430928 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:29.430964 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:29.430928 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:29.431784 7101 server_base.cc:1034] running on GCE node
I20260924 12:46:29.432524 7101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:29.433626 7101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:29.434849 7101 hybrid_clock.cc:648] HybridClock initialized: now 1790253989434797 us; error 45 us; skew 500 ppm
I20260924 12:46:29.435178 7101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:29.437661 7101 webserver.cc:460] Webserver started at http://127.2.81.65:46775/ using document root <none> and password file <none>
I20260924 12:46:29.438402 7101 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:29.438499 7101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:29.438788 7101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:29.441898 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "85e225dee2fe465badb929c2f732f555"
format_stamp: "Formatted at 2026-09-24 12:46:29 on dist-test-slave-9tm1"
server_key: "7cf0b2bf60859b2207e49664db1fee63"
server_key_iv: "36d5fa3bcfa5fb19a2dd04e5e7d1489c"
server_key_version: "encryptionkey@0"
I20260924 12:46:29.442764 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "85e225dee2fe465badb929c2f732f555"
format_stamp: "Formatted at 2026-09-24 12:46:29 on dist-test-slave-9tm1"
server_key: "7cf0b2bf60859b2207e49664db1fee63"
server_key_iv: "36d5fa3bcfa5fb19a2dd04e5e7d1489c"
server_key_version: "encryptionkey@0"
I20260924 12:46:29.448776 7101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 12:46:29.452914 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:29.454733 7101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:46:29.454933 7101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "85e225dee2fe465badb929c2f732f555"
format_stamp: "Formatted at 2026-09-24 12:46:29 on dist-test-slave-9tm1"
server_key: "7cf0b2bf60859b2207e49664db1fee63"
server_key_iv: "36d5fa3bcfa5fb19a2dd04e5e7d1489c"
server_key_version: "encryptionkey@0"
I20260924 12:46:29.455180 7101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:29.461584 7101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:29.462486 7101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:29.462774 7101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:29.463810 7101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:29.465335 7101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:29.465430 7101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:29.465544 7101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:29.465612 7101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:29.498561 7101 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:42447
I20260924 12:46:29.498648 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:42447 every 8 connection(s)
I20260924 12:46:29.499853 7101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:29.504163 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7101
I20260924 12:46:29.504365 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:29.505174 2373 external_mini_cluster.cc:1428] Setting key 56da98954aafb1082dcebc4ef135c449
I20260924 12:46:29.508539 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33103
--builtin_ntp_servers=127.2.81.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 12:46:29.510342 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:29.502763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:46103 (local address 127.2.81.126:33103)
0924 12:46:29.503232 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:29.503236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:29.504807 (+ 1571us) server_negotiation.cc:407] Connection header received
0924 12:46:29.507297 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:29.507303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:29.507370 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:29.507488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:29.509576 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:29.509582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:29.509586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:29.509609 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:29.509655 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:29.509669 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:29.509788 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:29.509984 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 12:46:29.510055 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260924 12:46:29.511495 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:29.503194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33103 (local address 127.2.81.65:46103)
0924 12:46:29.504524 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:29.504587 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:29.506802 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:29.508012 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:29.508028 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:29.508176 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:29.508911 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:29.508913 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:29.509136 (+ 223us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:29.509326 (+ 190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:29.509759 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:29.509771 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:29.509780 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:46:29.510071 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 12:46:29.510407 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":465,"threads_started":1}
I20260924 12:46:29.513500 7228 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33103
I20260924 12:46:29.513885 7228 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:29.514637 7228 heartbeater.cc:511] Master 127.2.81.126:33103 requested a full tablet report, sending...
I20260924 12:46:29.517175 7042 ts_manager.cc:194] Registered new tserver with Master: 85e225dee2fe465badb929c2f732f555 (127.2.81.65:42447)
W20260924 12:46:29.855379 7232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:29.855690 7232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:29.855809 7232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:29.864787 7232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:29.865039 7232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:29.865257 7232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:29.874950 7232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:29.876282 7232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:29.878500 7232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:29.892925 7237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:29.893007 7238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:29.893028 7240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:29.893627 7232 server_base.cc:1034] running on GCE node
I20260924 12:46:29.894404 7232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:29.895534 7232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:29.896781 7232 hybrid_clock.cc:648] HybridClock initialized: now 1790253989896769 us; error 77 us; skew 500 ppm
I20260924 12:46:29.897204 7232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:29.899889 7232 webserver.cc:460] Webserver started at http://127.2.81.66:36463/ using document root <none> and password file <none>
I20260924 12:46:29.900607 7232 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:29.900712 7232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:29.901067 7232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:29.904013 7232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c"
format_stamp: "Formatted at 2026-09-24 12:46:29 on dist-test-slave-9tm1"
server_key: "9ecf4b42d627ddc1acd8beebe15b2adb"
server_key_iv: "0696b279e317d039ba3ff27c8bc85915"
server_key_version: "encryptionkey@0"
I20260924 12:46:29.904839 7232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c"
format_stamp: "Formatted at 2026-09-24 12:46:29 on dist-test-slave-9tm1"
server_key: "9ecf4b42d627ddc1acd8beebe15b2adb"
server_key_iv: "0696b279e317d039ba3ff27c8bc85915"
server_key_version: "encryptionkey@0"
I20260924 12:46:29.910673 7232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 12:46:29.914337 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:29.915719 7232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:29.915874 7232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c"
format_stamp: "Formatted at 2026-09-24 12:46:29 on dist-test-slave-9tm1"
server_key: "9ecf4b42d627ddc1acd8beebe15b2adb"
server_key_iv: "0696b279e317d039ba3ff27c8bc85915"
server_key_version: "encryptionkey@0"
I20260924 12:46:29.916095 7232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:29.921633 7232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:29.922401 7232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:29.922654 7232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:29.923537 7232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:29.924883 7232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:29.924991 7232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:29.925097 7232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:29.925166 7232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:29.955579 7232 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:45173
I20260924 12:46:29.955629 7358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:45173 every 8 connection(s)
I20260924 12:46:29.957206 7232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:29.961016 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7232
I20260924 12:46:29.961197 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:29.961771 2373 external_mini_cluster.cc:1428] Setting key b4e56168fc0df7eb86f294c1cb7100f1
I20260924 12:46:29.965095 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33103
--builtin_ntp_servers=127.2.81.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 12:46:29.968811 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:29.961272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44619 (local address 127.2.81.126:33103)
0924 12:46:29.961460 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:29.961467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:29.963530 (+ 2063us) server_negotiation.cc:407] Connection header received
0924 12:46:29.965975 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:29.965982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:29.966056 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:29.966126 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:29.968025 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:29.968032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:29.968038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:29.968060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:29.968099 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:29.968114 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:29.968236 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:29.968418 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 12:46:29.968479 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:46:29.969976 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:29.960918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33103 (local address 127.2.81.66:44619)
0924 12:46:29.963373 (+ 2455us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:29.963426 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:29.965585 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:29.966695 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:29.966709 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:29.966854 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:29.967665 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:29.967669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:29.967719 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:29.967830 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:29.968216 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:29.968224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:29.968231 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:46:29.968492 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 12:46:29.968777 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1792,"thread_start_us":1493,"threads_started":1}
I20260924 12:46:29.974443 7359 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33103
I20260924 12:46:29.974771 7359 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:29.975376 7359 heartbeater.cc:511] Master 127.2.81.126:33103 requested a full tablet report, sending...
I20260924 12:46:29.976827 7042 ts_manager.cc:194] Registered new tserver with Master: 9bf65cd7113a46ca9a3fc5012fbfe68c (127.2.81.66:45173)
W20260924 12:46:30.322221 7362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:30.322563 7362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:30.322705 7362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:30.330595 7362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:30.330724 7362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:30.330880 7362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:30.340387 7362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:30.341764 7362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:30.343550 7362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:30.357486 7370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:30.357520 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:30.357688 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:30.358263 7362 server_base.cc:1034] running on GCE node
I20260924 12:46:30.358922 7362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:30.360096 7362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:30.361372 7362 hybrid_clock.cc:648] HybridClock initialized: now 1790253990361303 us; error 67 us; skew 500 ppm
I20260924 12:46:30.361717 7362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:30.364061 7362 webserver.cc:460] Webserver started at http://127.2.81.67:37509/ using document root <none> and password file <none>
I20260924 12:46:30.364795 7362 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:30.364885 7362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:30.365223 7362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:30.368391 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "58e02dcc74b249659b824bbec147595c"
format_stamp: "Formatted at 2026-09-24 12:46:30 on dist-test-slave-9tm1"
server_key: "61502334dd344cb96c167cdc03bdc5a3"
server_key_iv: "5b0f74242824257be41201dbe87a616c"
server_key_version: "encryptionkey@0"
I20260924 12:46:30.369324 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "58e02dcc74b249659b824bbec147595c"
format_stamp: "Formatted at 2026-09-24 12:46:30 on dist-test-slave-9tm1"
server_key: "61502334dd344cb96c167cdc03bdc5a3"
server_key_iv: "5b0f74242824257be41201dbe87a616c"
server_key_version: "encryptionkey@0"
I20260924 12:46:30.374783 7362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 12:46:30.378389 7376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:30.379863 7362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:30.380050 7362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "58e02dcc74b249659b824bbec147595c"
format_stamp: "Formatted at 2026-09-24 12:46:30 on dist-test-slave-9tm1"
server_key: "61502334dd344cb96c167cdc03bdc5a3"
server_key_iv: "5b0f74242824257be41201dbe87a616c"
server_key_version: "encryptionkey@0"
I20260924 12:46:30.380283 7362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:30.386309 7362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:30.387225 7362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:30.387499 7362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:30.388473 7362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:30.389955 7362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:30.390041 7362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:30.390147 7362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:30.390206 7362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:30.421726 7362 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:38689
I20260924 12:46:30.421784 7488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:38689 every 8 connection(s)
I20260924 12:46:30.423013 7362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:30.429303 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7362
I20260924 12:46:30.429483 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:30.430007 2373 external_mini_cluster.cc:1428] Setting key 4b7a091ef71e6693463c56f62997ef89
I20260924 12:46:30.431852 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.425516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:43371 (local address 127.2.81.126:33103)
0924 12:46:30.425725 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.425732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.427258 (+ 1526us) server_negotiation.cc:407] Connection header received
0924 12:46:30.429147 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.429153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.429240 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.429359 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.431055 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.431062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.431067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.431086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.431122 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.431137 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.431251 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.431500 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.431578 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:46:30.433347 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.426099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33103 (local address 127.2.81.67:43371)
0924 12:46:30.427066 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:30.427130 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:30.428747 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:30.429654 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:30.429667 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:30.429801 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:30.430658 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:30.430661 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:30.430718 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:30.430875 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:30.431266 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:30.431277 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:30.431286 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:46:30.431574 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 12:46:30.431904 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":282,"threads_started":1}
I20260924 12:46:30.434947 7489 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33103
I20260924 12:46:30.435353 7489 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:30.436007 7489 heartbeater.cc:511] Master 127.2.81.126:33103 requested a full tablet report, sending...
I20260924 12:46:30.437582 7042 ts_manager.cc:194] Registered new tserver with Master: 58e02dcc74b249659b824bbec147595c (127.2.81.67:38689)
I20260924 12:46:30.446266 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:46:30.455547 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.451255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.2.81.126:33103)
0924 12:46:30.451424 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.451429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.451881 (+ 452us) server_negotiation.cc:407] Connection header received
0924 12:46:30.451955 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.451960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.452020 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.452340 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.453882 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.453887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.453891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.453909 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.453946 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.453959 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.454214 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.454578 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.454634 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:46:30.461349 7042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:30.468631 7042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:30.499759 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.483003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.2.81.65:42447)
0924 12:46:30.497173 (+ 14170us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.497181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.497210 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:46:30.497281 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.497288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.497495 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.497743 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.498746 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.498754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.498766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.498783 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.498846 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.498872 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.499019 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.499267 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.499369 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14016,"thread_start_us":13860,"threads_started":1}
I20260924 12:46:30.499773 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.485090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45173 (local address 127.0.0.1:35986)
0924 12:46:30.495437 (+ 10347us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:30.495456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:30.495552 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:30.497216 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:30.497221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:30.497245 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:30.498553 (+ 1308us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:30.498556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:30.498569 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:30.498602 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:30.499054 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:30.499065 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:30.499073 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:46:30.499254 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 12:46:30.499362 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10190,"thread_start_us":8957,"threads_started":1}
I20260924 12:46:30.499830 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.484290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.2.81.67:38689)
0924 12:46:30.495152 (+ 10862us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.495160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.495238 (+ 78us) server_negotiation.cc:407] Connection header received
0924 12:46:30.495319 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.495327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.495615 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.495784 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.498848 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.498856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.498866 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.498887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.498939 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.498959 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.499125 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.499399 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.499489 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10685,"thread_start_us":12071,"threads_started":1}
I20260924 12:46:30.500674 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.482808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34114)
0924 12:46:30.494679 (+ 11871us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:30.494878 (+ 199us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:30.496629 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:30.497671 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:30.497676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:30.497697 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:30.498371 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:30.498376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:30.498408 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:30.498497 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:30.499079 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:30.499084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:30.499088 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 12:46:30.500188 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0924 12:46:30.500242 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11580,"thread_start_us":985,"threads_started":1}
I20260924 12:46:30.500846 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.483571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:38689 (local address 127.0.0.1:38302)
0924 12:46:30.494976 (+ 11405us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:30.494997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:30.495164 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:30.497070 (+ 1906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:30.497078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:30.497187 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:30.498371 (+ 1184us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:30.498376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:30.498408 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:30.498696 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:30.499172 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:30.499176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:30.499181 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 12:46:30.499300 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 12:46:30.500396 (+ 1096us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11238,"thread_start_us":10100,"threads_started":1}
I20260924 12:46:30.504427 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.486068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.2.81.66:45173)
0924 12:46:30.494604 (+ 8536us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.494617 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.495670 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 12:46:30.495868 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.495879 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.496154 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.496327 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.498736 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.498746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.498762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.498792 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.498851 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.498872 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.499000 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.500232 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.501777 (+ 1545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8327,"thread_start_us":8208,"threads_started":1}
I20260924 12:46:30.504467 7163 tablet_service.cc:1467] Processing CreateTablet for tablet be2c54f807df4ccc87b344daec2c135c (DEFAULT_TABLE table=test-table [id=97e40703d1ee47eca2c97c0fd0e789de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:30.504765 7424 tablet_service.cc:1467] Processing CreateTablet for tablet be2c54f807df4ccc87b344daec2c135c (DEFAULT_TABLE table=test-table [id=97e40703d1ee47eca2c97c0fd0e789de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:30.506075 7163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2c54f807df4ccc87b344daec2c135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:30.506971 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2c54f807df4ccc87b344daec2c135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:30.508582 7294 tablet_service.cc:1467] Processing CreateTablet for tablet be2c54f807df4ccc87b344daec2c135c (DEFAULT_TABLE table=test-table [id=97e40703d1ee47eca2c97c0fd0e789de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:30.510066 7294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2c54f807df4ccc87b344daec2c135c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:30.517434 7507 tablet_bootstrap.cc:492] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: Bootstrap starting.
I20260924 12:46:30.520658 7228 heartbeater.cc:503] Master 127.2.81.126:33103 was elected leader, sending a full tablet report...
I20260924 12:46:30.520776 7507 tablet_bootstrap.cc:654] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:30.522020 7507 log.cc:824] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:30.526894 7510 tablet_bootstrap.cc:492] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Bootstrap starting.
I20260924 12:46:30.527668 7508 tablet_bootstrap.cc:492] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Bootstrap starting.
I20260924 12:46:30.530849 7510 tablet_bootstrap.cc:654] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:30.530889 7508 tablet_bootstrap.cc:654] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:30.531816 7508 log.cc:824] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:30.531852 7507 tablet_bootstrap.cc:492] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: No bootstrap required, opened a new log
I20260924 12:46:30.532092 7510 log.cc:824] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:30.532125 7507 ts_tablet_manager.cc:1397] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260924 12:46:30.534786 7508 tablet_bootstrap.cc:492] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: No bootstrap required, opened a new log
I20260924 12:46:30.535120 7508 ts_tablet_manager.cc:1397] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260924 12:46:30.535265 7510 tablet_bootstrap.cc:492] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: No bootstrap required, opened a new log
I20260924 12:46:30.535969 7510 ts_tablet_manager.cc:1397] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 12:46:30.542061 7508 raft_consensus.cc:348] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.542104 7507 raft_consensus.cc:348] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.542446 7508 raft_consensus.cc:374] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:30.542565 7508 raft_consensus.cc:729] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e02dcc74b249659b824bbec147595c, State: Initialized, Role: FOLLOWER
I20260924 12:46:30.542567 7507 raft_consensus.cc:374] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:30.542660 7507 raft_consensus.cc:729] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e225dee2fe465badb929c2f732f555, State: Initialized, Role: FOLLOWER
I20260924 12:46:30.543144 7508 consensus_queue.cc:260] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.543258 7507 consensus_queue.cc:260] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.544202 7489 heartbeater.cc:503] Master 127.2.81.126:33103 was elected leader, sending a full tablet report...
I20260924 12:46:30.545367 7507 ts_tablet_manager.cc:1428] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260924 12:46:30.545426 7508 ts_tablet_manager.cc:1428] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 12:46:30.547307 7510 raft_consensus.cc:348] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.547832 7510 raft_consensus.cc:374] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:30.547972 7510 raft_consensus.cc:729] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf65cd7113a46ca9a3fc5012fbfe68c, State: Initialized, Role: FOLLOWER
I20260924 12:46:30.549069 7510 consensus_queue.cc:260] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.550084 7359 heartbeater.cc:503] Master 127.2.81.126:33103 was elected leader, sending a full tablet report...
I20260924 12:46:30.550518 7510 ts_tablet_manager.cc:1428] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260924 12:46:30.674715 7490 tablet.cc:2369] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:30.709662 7360 tablet.cc:2369] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:30.752609 7229 tablet.cc:2369] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:30.842233 7514 raft_consensus.cc:482] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:30.842532 7514 raft_consensus.cc:504] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.844250 7514 leader_election.cc:288] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf65cd7113a46ca9a3fc5012fbfe68c (127.2.81.66:45173), 58e02dcc74b249659b824bbec147595c (127.2.81.67:38689)
I20260924 12:46:30.847605 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.844783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:60939 (local address 127.2.81.67:38689)
0924 12:46:30.845006 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.845013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.845708 (+ 695us) server_negotiation.cc:407] Connection header received
0924 12:46:30.845851 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.845857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.845904 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.845988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.847056 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.847063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.847066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.847082 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.847113 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.847125 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.847259 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.847359 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.847414 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:46:30.847723 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.844820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:42067 (local address 127.2.81.66:45173)
0924 12:46:30.844997 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.845003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.845572 (+ 569us) server_negotiation.cc:407] Connection header received
0924 12:46:30.845782 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.845789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.845853 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.845941 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.847073 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.847078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.847081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.847097 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.847127 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.847138 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.847255 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.847402 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.847460 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 12:46:30.847898 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.844473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45173 (local address 127.2.81.65:42067)
0924 12:46:30.845449 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:30.845478 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:30.845626 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:30.845979 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:30.845984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:30.846006 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:30.846869 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:30.846873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:30.846890 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:30.846933 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:30.847224 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:30.847229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:30.847234 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 12:46:30.847410 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 12:46:30.847473 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":617,"threads_started":1}
I20260924 12:46:30.848330 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.844736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:38689 (local address 127.2.81.65:60939)
0924 12:46:30.845608 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:30.845623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:30.845731 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:30.846025 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:30.846030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:30.846052 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:30.846803 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:46:30.846805 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:30.846823 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:46:30.846886 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:30.847224 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:30.847229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:30.847234 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 12:46:30.847915 (+ 681us) client_negotiation.cc:240] Negotiation successful
0924 12:46:30.847968 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":500,"threads_started":1}
I20260924 12:46:30.849015 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2c54f807df4ccc87b344daec2c135c" candidate_uuid: "85e225dee2fe465badb929c2f732f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" is_pre_election: true
I20260924 12:46:30.849328 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2c54f807df4ccc87b344daec2c135c" candidate_uuid: "85e225dee2fe465badb929c2f732f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e02dcc74b249659b824bbec147595c" is_pre_election: true
I20260924 12:46:30.849534 7314 raft_consensus.cc:2454] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e225dee2fe465badb929c2f732f555 in term 0.
I20260924 12:46:30.849895 7444 raft_consensus.cc:2454] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e225dee2fe465badb929c2f732f555 in term 0.
I20260924 12:46:30.850191 7119 leader_election.cc:302] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e225dee2fe465badb929c2f732f555, 9bf65cd7113a46ca9a3fc5012fbfe68c; no voters:
I20260924 12:46:30.850812 7514 raft_consensus.cc:2781] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:30.850929 7514 raft_consensus.cc:482] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:30.850996 7514 raft_consensus.cc:3037] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:30.853324 7514 raft_consensus.cc:504] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.853930 7514 leader_election.cc:288] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf65cd7113a46ca9a3fc5012fbfe68c (127.2.81.66:45173), 58e02dcc74b249659b824bbec147595c (127.2.81.67:38689)
I20260924 12:46:30.854501 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2c54f807df4ccc87b344daec2c135c" candidate_uuid: "85e225dee2fe465badb929c2f732f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c"
I20260924 12:46:30.854506 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2c54f807df4ccc87b344daec2c135c" candidate_uuid: "85e225dee2fe465badb929c2f732f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e02dcc74b249659b824bbec147595c"
I20260924 12:46:30.854697 7314 raft_consensus.cc:3037] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:30.854700 7444 raft_consensus.cc:3037] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:30.856933 7444 raft_consensus.cc:2454] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e225dee2fe465badb929c2f732f555 in term 1.
I20260924 12:46:30.857335 7314 raft_consensus.cc:2454] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e225dee2fe465badb929c2f732f555 in term 1.
I20260924 12:46:30.857478 7117 leader_election.cc:302] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e02dcc74b249659b824bbec147595c, 85e225dee2fe465badb929c2f732f555; no voters:
I20260924 12:46:30.857820 7514 raft_consensus.cc:2781] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:30.858273 7514 raft_consensus.cc:686] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 1 LEADER]: Becoming Leader. State: Replica: 85e225dee2fe465badb929c2f732f555, State: Running, Role: LEADER
I20260924 12:46:30.858744 7514 consensus_queue.cc:237] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } }
I20260924 12:46:30.863306 7042 catalog_manager.cc:5495] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e225dee2fe465badb929c2f732f555 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "85e225dee2fe465badb929c2f732f555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e225dee2fe465badb929c2f732f555" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 42447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:30.947922 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:30.945866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.2.81.65:42447)
0924 12:46:30.946025 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:30.946030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:30.946048 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:30.946140 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:30.946144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:30.946254 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:30.946346 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:30.947315 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:30.947320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:30.947323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:30.947339 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:30.947371 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:30.947383 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:30.947484 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:30.947591 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 12:46:30.947653 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 12:46:30.957147 7444 raft_consensus.cc:1261] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 1 FOLLOWER]: Refusing update from remote peer 85e225dee2fe465badb929c2f732f555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:30.957147 7314 raft_consensus.cc:1261] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 1 FOLLOWER]: Refusing update from remote peer 85e225dee2fe465badb929c2f732f555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:30.958408 7518 consensus_queue.cc:1035] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf65cd7113a46ca9a3fc5012fbfe68c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:30.958839 7514 consensus_queue.cc:1035] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e02dcc74b249659b824bbec147595c" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:30.975246 7521 mvcc.cc:205] Tried to move back new op lower bound from 7332880346947870720 to 7332880346561183744. Current Snapshot: MvccSnapshot[applied={T|T < 7332880346947870720}]
I20260924 12:46:31.001121 7042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:31.001366 7042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:31.004844 7042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5767121ad344ec9a10e79caddfc6a63: Sending DeleteTablet for 3 replicas of tablet be2c54f807df4ccc87b344daec2c135c
I20260924 12:46:31.006305 7163 tablet_service.cc:1514] Processing DeleteTablet for tablet be2c54f807df4ccc87b344daec2c135c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:30 UTC) from {username='slave'} at 127.0.0.1:34114
I20260924 12:46:31.006125 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet be2c54f807df4ccc87b344daec2c135c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:30 UTC) from {username='slave'} at 127.0.0.1:38302
I20260924 12:46:31.006763 7294 tablet_service.cc:1514] Processing DeleteTablet for tablet be2c54f807df4ccc87b344daec2c135c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:30 UTC) from {username='slave'} at 127.0.0.1:35986
I20260924 12:46:31.009116 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7101
I20260924 12:46:31.020785 7531 tablet_replica.cc:330] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555: stopping tablet replica
I20260924 12:46:31.021466 7533 tablet_replica.cc:330] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: stopping tablet replica
I20260924 12:46:31.021448 7531 raft_consensus.cc:2229] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:31.021729 7532 tablet_replica.cc:330] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: stopping tablet replica
I20260924 12:46:31.021801 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.010944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.2.81.65:42447)
0924 12:46:31.011141 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:31.011148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:31.016462 (+ 5314us) server_negotiation.cc:407] Connection header received
0924 12:46:31.017496 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:31.017503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:31.017576 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:31.017653 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:31.020689 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:31.020697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:31.020703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:31.020732 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:31.020783 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:31.020802 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:31.020927 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:31.021420 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 12:46:31.021497 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:46:31.021868 7533 raft_consensus.cc:2229] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:31.022188 7532 raft_consensus.cc:2229] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:31.022392 7533 raft_consensus.cc:2258] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:31.022680 7532 raft_consensus.cc:2258] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:31.022671 7531 raft_consensus.cc:2258] T be2c54f807df4ccc87b344daec2c135c P 85e225dee2fe465badb929c2f732f555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:31.024561 7533 ts_tablet_manager.cc:1905] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:31.024722 7532 ts_tablet_manager.cc:1905] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:31.027619 7223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:31.031085 7532 ts_tablet_manager.cc:1918] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:31.031275 7533 ts_tablet_manager.cc:1918] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:31.031371 7532 log.cc:1196] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/be2c54f807df4ccc87b344daec2c135c
I20260924 12:46:31.031435 7533 log.cc:1196] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/be2c54f807df4ccc87b344daec2c135c
I20260924 12:46:31.031993 7532 ts_tablet_manager.cc:1939] T be2c54f807df4ccc87b344daec2c135c P 58e02dcc74b249659b824bbec147595c: Deleting consensus metadata
I20260924 12:46:31.032171 7533 ts_tablet_manager.cc:1939] T be2c54f807df4ccc87b344daec2c135c P 9bf65cd7113a46ca9a3fc5012fbfe68c: Deleting consensus metadata
I20260924 12:46:31.033751 7027 catalog_manager.cc:4841] TS 58e02dcc74b249659b824bbec147595c (127.2.81.67:38689): tablet be2c54f807df4ccc87b344daec2c135c (table test-table [id=97e40703d1ee47eca2c97c0fd0e789de]) successfully deleted
I20260924 12:46:31.033857 7029 catalog_manager.cc:4841] TS 9bf65cd7113a46ca9a3fc5012fbfe68c (127.2.81.66:45173): tablet be2c54f807df4ccc87b344daec2c135c (table test-table [id=97e40703d1ee47eca2c97c0fd0e789de]) successfully deleted
I20260924 12:46:31.122939 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7101
W20260924 12:46:31.137465 7029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:42447 (error 108)
I20260924 12:46:31.137743 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7232
I20260924 12:46:31.139366 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.138594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34138)
0924 12:46:31.139011 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.139152 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":237,"threads_started":1}
W20260924 12:46:31.139658 7029 catalog_manager.cc:4584] TS 85e225dee2fe465badb929c2f732f555 (127.2.81.65:42447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be2c54f807df4ccc87b344daec2c135c: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
I20260924 12:46:31.140733 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.138571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.2.81.66:45173)
0924 12:46:31.138761 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:31.138768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:31.138791 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:46:31.138862 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:31.138867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:31.138968 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:31.139060 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:31.140151 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:31.140155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:31.140158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:31.140175 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:31.140206 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:31.140218 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:31.140329 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:31.140430 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 12:46:31.140485 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:46:31.141351 7354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:31.193648 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.193110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34140)
0924 12:46:31.193345 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.193474 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 12:46:31.194849 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.194395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34152)
0924 12:46:31.194557 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.194663 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 12:46:31.231139 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7232
I20260924 12:46:31.243590 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7362
I20260924 12:46:31.246362 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.244345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.2.81.67:38689)
0924 12:46:31.244489 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:31.244494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:31.244512 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:31.244569 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:31.244573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:31.244785 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:31.244905 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:31.245837 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:31.245842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:31.245845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:31.245861 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:31.245892 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:31.245903 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:31.246009 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:31.246067 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 12:46:31.246116 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 12:46:31.246973 7484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:31.256069 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.255524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34168)
0924 12:46:31.255775 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.255889 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 12:46:31.257177 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.256718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34170)
0924 12:46:31.256874 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.256997 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 12:46:31.337117 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7362
I20260924 12:46:31.346175 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.345580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34180)
0924 12:46:31.345887 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.345993 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 12:46:31.347446 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:31.346964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:42447 (local address 127.0.0.1:34194)
0924 12:46:31.347170 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:31.347260 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:42447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 12:46:31.350330 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7011
I20260924 12:46:31.351454 7072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:31.458976 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7011
2026-09-24T12:46:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2951 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 12:46:31.498431 2373 mini_oidc.cc:152] Starting JWKS server
I20260924 12:46:31.501543 2373 webserver.cc:460] Webserver started at https://127.0.0.1:37101/ using document root <none> and password file <none>
I20260924 12:46:31.502072 2373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 12:46:31.502875 2373 webserver.cc:460] Webserver started at http://127.0.0.1:36655/ using document root <none> and password file <none>
2026-09-24T12:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:31Z Disabled control of system clock
I20260924 12:46:31.534588 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:43369
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:43369
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37101/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/testchainca.cert with env {}
W20260924 12:46:31.895094 7546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:31.895423 7546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:31.895560 7546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:31.903579 7546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:31.903734 7546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:31.903800 7546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:31.903877 7546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:31.903934 7546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 12:46:31.904016 7546 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 12:46:31.904065 7546 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37101/jwks.json
I20260924 12:46:31.915171 7546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:43369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37101/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:31.916502 7546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:31.917860 7546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:31.929840 7551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:31.929885 7552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:31.929745 7554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:31.930727 7546 server_base.cc:1034] running on GCE node
I20260924 12:46:31.931524 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:31.933074 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:31.934310 7546 hybrid_clock.cc:648] HybridClock initialized: now 1790253991934236 us; error 68 us; skew 500 ppm
I20260924 12:46:31.934725 7546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:31.941262 7546 webserver.cc:460] Webserver started at http://127.2.81.126:41291/ using document root <none> and password file <none>
I20260924 12:46:31.942080 7546 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:31.942183 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:31.942492 7546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:31.945614 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "3a9baa0e91c54a18842f9ae235464935"
format_stamp: "Formatted at 2026-09-24 12:46:31 on dist-test-slave-9tm1"
server_key: "52862fe56a661265beff9ed0308c886c"
server_key_iv: "6868429855fb6181b3a047a41625b9cf"
server_key_version: "encryptionkey@0"
I20260924 12:46:31.946451 7546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "3a9baa0e91c54a18842f9ae235464935"
format_stamp: "Formatted at 2026-09-24 12:46:31 on dist-test-slave-9tm1"
server_key: "52862fe56a661265beff9ed0308c886c"
server_key_iv: "6868429855fb6181b3a047a41625b9cf"
server_key_version: "encryptionkey@0"
I20260924 12:46:31.952198 7546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 12:46:31.956008 7560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:31.957635 7546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:46:31.957834 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "3a9baa0e91c54a18842f9ae235464935"
format_stamp: "Formatted at 2026-09-24 12:46:31 on dist-test-slave-9tm1"
server_key: "52862fe56a661265beff9ed0308c886c"
server_key_iv: "6868429855fb6181b3a047a41625b9cf"
server_key_version: "encryptionkey@0"
I20260924 12:46:31.958052 7546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:31.989071 7546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:31.989931 7546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:31.990195 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:32.008808 7546 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:43369
I20260924 12:46:32.008839 7611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:43369 every 8 connection(s)
I20260924 12:46:32.010198 7546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:32.014418 7612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:32.017270 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7546
I20260924 12:46:32.017493 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:32.018241 2373 external_mini_cluster.cc:1428] Setting key 78ac05cf404c384f94d5b4fa1aa6a246
I20260924 12:46:32.027700 7612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935: Bootstrap starting.
I20260924 12:46:32.031399 7612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:32.032476 7612 log.cc:824] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:32.033391 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:32.020454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.2.81.126:43369)
0924 12:46:32.021531 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:32.021559 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:32.021605 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:46:32.023339 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:32.023368 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:32.023929 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:32.024309 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:32.025079 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.026512 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.028366 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.029206 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.029363 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:32.030790 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:32.030799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:32.030814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:32.031379 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:32.031477 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:32.031508 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:32.031690 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:32.031903 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 12:46:32.032206 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":352,"threads_started":1}
I20260924 12:46:32.035977 7612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935: No bootstrap required, opened a new log
I20260924 12:46:32.037833 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:46:32.044672 7612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9baa0e91c54a18842f9ae235464935" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43369 } }
I20260924 12:46:32.045156 7612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:32.045251 7612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9baa0e91c54a18842f9ae235464935, State: Initialized, Role: FOLLOWER
I20260924 12:46:32.045980 7612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9baa0e91c54a18842f9ae235464935" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43369 } }
I20260924 12:46:32.046273 7612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:32.046429 7612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:32.046622 7612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:32.049039 7612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9baa0e91c54a18842f9ae235464935" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43369 } }
I20260924 12:46:32.049605 7612 leader_election.cc:302] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a9baa0e91c54a18842f9ae235464935; no voters:
I20260924 12:46:32.050196 7612 leader_election.cc:288] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:32.050424 7617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:32.051152 7617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9baa0e91c54a18842f9ae235464935, State: Running, Role: LEADER
I20260924 12:46:32.051726 7617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9baa0e91c54a18842f9ae235464935" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43369 } }
I20260924 12:46:32.052268 7612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:32.054526 7619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9baa0e91c54a18842f9ae235464935. Latest consensus state: current_term: 1 leader_uuid: "3a9baa0e91c54a18842f9ae235464935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9baa0e91c54a18842f9ae235464935" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43369 } } }
I20260924 12:46:32.054776 7618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a9baa0e91c54a18842f9ae235464935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9baa0e91c54a18842f9ae235464935" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43369 } } }
I20260924 12:46:32.055094 7619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:32.055143 7618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:32.059469 7623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:32.067622 7623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:32.078698 7623 catalog_manager.cc:1347] Generated new cluster ID: 542952c62be44dbcb90dede801c666eb
I20260924 12:46:32.078863 7623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:32.092096 7623 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:32.093508 7623 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:32.101289 7623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a9baa0e91c54a18842f9ae235464935: Generated new TSK 0
I20260924 12:46:32.102448 7623 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 12:46:32.142158 7642 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 12:46:32.142316 7642 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 12:46:32.143472 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:32.118964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.2.81.126:43369)
0924 12:46:32.119169 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:32.119176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:32.119788 (+ 612us) server_negotiation.cc:407] Connection header received
0924 12:46:32.120133 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:32.120139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:32.120240 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:32.120422 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0924 12:46:32.120925 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.122100 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.124545 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.125271 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.125417 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:32.125978 (+ 561us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 12:46:32.142787 (+ 16809us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 12:46:32.143175 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 12:46:32.143601 7615 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 12:46:32.146051 7641 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:32.118962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:43369 (local address 127.0.0.1:44178)
0924 12:46:32.119671 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:32.119692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:32.119854 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:32.120437 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:32.120441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:32.120451 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 12:46:32.120753 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:32.120761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.122373 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.122378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:32.124342 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:32.124366 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.125469 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.125474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:32.125612 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:32.125686 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 12:46:32.143120 (+ 17434us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 12:46:32.143396 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.81.126:43369: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":380,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.81.126:43369: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 12:46:32.147779 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7546
I20260924 12:46:32.149523 7607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:32.244252 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7546
2026-09-24T12:46:32Z chronyd exiting
I20260924 12:46:32.291203 2373 test_util.cc:175] -----------------------------------------------
I20260924 12:46:32.291361 2373 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790253949676453-2373-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (798 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 12:46:32.295686 2373 mini_oidc.cc:152] Starting JWKS server
I20260924 12:46:32.297662 2373 webserver.cc:460] Webserver started at https://127.0.0.1:45039/ using document root <none> and password file <none>
I20260924 12:46:32.297900 2373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 12:46:32.298578 2373 webserver.cc:460] Webserver started at http://127.0.0.1:38657/ using document root <none> and password file <none>
2026-09-24T12:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:32Z Disabled control of system clock
I20260924 12:46:32.324517 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41341
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:41341
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45039/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/testchainca.cert with env {}
W20260924 12:46:32.658151 7649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:32.658463 7649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:32.658591 7649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:32.666415 7649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:32.666553 7649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:32.666615 7649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:32.666673 7649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:32.666716 7649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 12:46:32.666775 7649 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 12:46:32.666811 7649 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45039/jwks.json
I20260924 12:46:32.676957 7649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:41341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45039/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:32.678246 7649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:32.679479 7649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:32.691675 7654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:32.691738 7657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:32.691636 7655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:32.692507 7649 server_base.cc:1034] running on GCE node
I20260924 12:46:32.693356 7649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:32.694973 7649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:32.696228 7649 hybrid_clock.cc:648] HybridClock initialized: now 1790253992696179 us; error 41 us; skew 500 ppm
I20260924 12:46:32.696611 7649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:32.699156 7649 webserver.cc:460] Webserver started at http://127.2.81.126:46487/ using document root <none> and password file <none>
I20260924 12:46:32.699865 7649 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:32.699971 7649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:32.700284 7649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:32.703269 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "bfaacc5bbcca463eaf8ee85f43afafad"
format_stamp: "Formatted at 2026-09-24 12:46:32 on dist-test-slave-9tm1"
server_key: "1783d16d2939c4657077f4b037f3403e"
server_key_iv: "f0b950b6aa40c715e1607b76fd3977ea"
server_key_version: "encryptionkey@0"
I20260924 12:46:32.704018 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "bfaacc5bbcca463eaf8ee85f43afafad"
format_stamp: "Formatted at 2026-09-24 12:46:32 on dist-test-slave-9tm1"
server_key: "1783d16d2939c4657077f4b037f3403e"
server_key_iv: "f0b950b6aa40c715e1607b76fd3977ea"
server_key_version: "encryptionkey@0"
I20260924 12:46:32.709416 7649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:46:32.713624 7663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:32.715564 7649 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:46:32.715775 7649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "bfaacc5bbcca463eaf8ee85f43afafad"
format_stamp: "Formatted at 2026-09-24 12:46:32 on dist-test-slave-9tm1"
server_key: "1783d16d2939c4657077f4b037f3403e"
server_key_iv: "f0b950b6aa40c715e1607b76fd3977ea"
server_key_version: "encryptionkey@0"
I20260924 12:46:32.716023 7649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:32.734959 7649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:32.736464 7649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:32.736749 7649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:32.756088 7649 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:41341
I20260924 12:46:32.756107 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:41341 every 8 connection(s)
I20260924 12:46:32.757572 7649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:32.761826 7715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:32.763823 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7649
I20260924 12:46:32.763978 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:32.764624 2373 external_mini_cluster.cc:1428] Setting key 3da9fb470313ee4f5a5dde9a1dd96a14
I20260924 12:46:32.775801 7715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad: Bootstrap starting.
I20260924 12:46:32.780169 7715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:32.781595 7715 log.cc:824] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:32.781924 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:32.766728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.2.81.126:41341)
0924 12:46:32.767889 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:32.767906 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:32.767961 (+ 55us) server_negotiation.cc:407] Connection header received
0924 12:46:32.770254 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:32.770285 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:32.771035 (+ 750us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:32.771617 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:32.772310 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.774387 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.775871 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.777070 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.777309 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:32.778599 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:32.778610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:32.778631 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:32.779488 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:32.779637 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:32.779673 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:32.779807 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:32.780122 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 12:46:32.780504 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":360,"threads_started":1}
I20260924 12:46:32.785043 7715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad: No bootstrap required, opened a new log
I20260924 12:46:32.786759 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:46:32.793735 7715 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41341 } }
I20260924 12:46:32.794124 7715 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:32.794203 7715 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaacc5bbcca463eaf8ee85f43afafad, State: Initialized, Role: FOLLOWER
I20260924 12:46:32.794727 7715 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41341 } }
I20260924 12:46:32.794931 7715 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:32.795071 7715 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:32.795248 7715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:32.797472 7715 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41341 } }
I20260924 12:46:32.797961 7715 leader_election.cc:302] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfaacc5bbcca463eaf8ee85f43afafad; no voters:
I20260924 12:46:32.798524 7715 leader_election.cc:288] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:32.798673 7720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:32.799381 7720 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [term 1 LEADER]: Becoming Leader. State: Replica: bfaacc5bbcca463eaf8ee85f43afafad, State: Running, Role: LEADER
I20260924 12:46:32.799878 7720 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41341 } }
I20260924 12:46:32.800531 7715 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:32.802419 7722 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfaacc5bbcca463eaf8ee85f43afafad. Latest consensus state: current_term: 1 leader_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41341 } } }
I20260924 12:46:32.802444 7721 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfaacc5bbcca463eaf8ee85f43afafad" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41341 } } }
I20260924 12:46:32.802862 7722 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:32.802997 7721 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:32.803752 7725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:32.808894 7725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:32.818645 7725 catalog_manager.cc:1347] Generated new cluster ID: 268b015d0ef24dfc8aa9312aabcd8485
I20260924 12:46:32.818776 7725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:32.826160 7725 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:32.827502 7725 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:32.843151 7725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfaacc5bbcca463eaf8ee85f43afafad: Generated new TSK 0
I20260924 12:46:32.844048 7725 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 12:46:32.883659 7745 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 12:46:32.883782 7745 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 12:46:32.885073 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:32.864730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.2.81.126:41341)
0924 12:46:32.864931 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:32.864938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:32.865361 (+ 423us) server_negotiation.cc:407] Connection header received
0924 12:46:32.865567 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:32.865573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:32.865687 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:32.865853 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0924 12:46:32.866286 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.867426 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.868815 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.869520 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.869658 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:32.870177 (+ 519us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 12:46:32.884299 (+ 14122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 12:46:32.884705 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56508: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 12:46:32.885205 7718 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56508: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 12:46:32.885474 7744 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:32.864553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41341 (local address 127.0.0.1:56508)
0924 12:46:32.865266 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:32.865283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:32.865404 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:32.865883 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:32.865887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:32.865895 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 12:46:32.866155 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:32.866163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.867593 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.867597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:32.868670 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:32.868682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:32.869702 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:32.869707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:32.869861 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:32.869899 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 12:46:32.884574 (+ 14675us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 12:46:32.884780 (+ 206us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.81.126:41341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":414,"threads_started":1}
I20260924 12:46:32.887291 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7649
I20260924 12:46:32.888391 7710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:32.980374 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7649
2026-09-24T12:46:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 12:46:33.034366 2373 mini_oidc.cc:152] Starting JWKS server
I20260924 12:46:33.036382 2373 webserver.cc:460] Webserver started at https://127.0.0.1:44777/ using document root <none> and password file <none>
I20260924 12:46:33.036634 2373 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 12:46:33.037384 2373 webserver.cc:460] Webserver started at http://127.0.0.1:43665/ using document root <none> and password file <none>
2026-09-24T12:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:33Z Disabled control of system clock
I20260924 12:46:33.065322 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:43609
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:43609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44777/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 12:46:33.433542 7752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:33.433871 7752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:33.433957 7752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:33.441726 7752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:33.441870 7752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:33.441920 7752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:33.441977 7752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:33.442020 7752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 12:46:33.442080 7752 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 12:46:33.442117 7752 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44777/jwks.json
I20260924 12:46:33.454238 7752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:43609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44777/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:33.455570 7752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:33.457888 7752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:33.473660 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:33.473665 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:33.473835 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:33.474682 7752 server_base.cc:1034] running on GCE node
I20260924 12:46:33.475560 7752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:33.477313 7752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:33.478621 7752 hybrid_clock.cc:648] HybridClock initialized: now 1790253993478508 us; error 64 us; skew 500 ppm
I20260924 12:46:33.479025 7752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:33.481747 7752 webserver.cc:460] Webserver started at http://127.2.81.126:44731/ using document root <none> and password file <none>
I20260924 12:46:33.482478 7752 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:33.482599 7752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:33.482914 7752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:33.485920 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "8d02ca9716994198899639ffc49f17df"
format_stamp: "Formatted at 2026-09-24 12:46:33 on dist-test-slave-9tm1"
server_key: "49adf0acc397c33f846ee25e370e82c9"
server_key_iv: "5b4b612b2155bc9b1e9032b4f70a39e6"
server_key_version: "encryptionkey@0"
I20260924 12:46:33.486717 7752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "8d02ca9716994198899639ffc49f17df"
format_stamp: "Formatted at 2026-09-24 12:46:33 on dist-test-slave-9tm1"
server_key: "49adf0acc397c33f846ee25e370e82c9"
server_key_iv: "5b4b612b2155bc9b1e9032b4f70a39e6"
server_key_version: "encryptionkey@0"
I20260924 12:46:33.493371 7752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:46:33.497026 7766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:33.498534 7752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:33.498709 7752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "8d02ca9716994198899639ffc49f17df"
format_stamp: "Formatted at 2026-09-24 12:46:33 on dist-test-slave-9tm1"
server_key: "49adf0acc397c33f846ee25e370e82c9"
server_key_iv: "5b4b612b2155bc9b1e9032b4f70a39e6"
server_key_version: "encryptionkey@0"
I20260924 12:46:33.498931 7752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:33.519881 7752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:33.520743 7752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:33.521034 7752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:33.538420 7752 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:43609
I20260924 12:46:33.538435 7817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:43609 every 8 connection(s)
I20260924 12:46:33.539814 7752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:33.541422 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7752
I20260924 12:46:33.541654 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:33.542423 2373 external_mini_cluster.cc:1428] Setting key 6387da86e9bde915ae44c8741d24a8e3
I20260924 12:46:33.544509 7818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:33.555454 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:33.544243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.2.81.126:43609)
0924 12:46:33.545302 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:33.545313 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:33.545353 (+ 40us) server_negotiation.cc:407] Connection header received
0924 12:46:33.546917 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:33.546939 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:33.547413 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:33.547867 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:33.548408 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:33.549835 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:33.551129 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:33.551809 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:33.551971 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:33.553124 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:33.553132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:33.553148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:33.553728 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:33.553822 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:33.553844 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:33.553951 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:33.554167 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 12:46:33.554482 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260924 12:46:33.555866 7818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df: Bootstrap starting.
I20260924 12:46:33.559429 7818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:33.559882 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:46:33.560797 7818 log.cc:824] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:33.563845 7818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df: No bootstrap required, opened a new log
I20260924 12:46:33.571472 7818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02ca9716994198899639ffc49f17df" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43609 } }
I20260924 12:46:33.571877 7818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:33.571956 7818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d02ca9716994198899639ffc49f17df, State: Initialized, Role: FOLLOWER
I20260924 12:46:33.572535 7818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02ca9716994198899639ffc49f17df" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43609 } }
I20260924 12:46:33.572770 7818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:33.572865 7818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:33.573007 7818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:33.575202 7818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02ca9716994198899639ffc49f17df" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43609 } }
I20260924 12:46:33.575726 7818 leader_election.cc:302] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d02ca9716994198899639ffc49f17df; no voters:
I20260924 12:46:33.576295 7818 leader_election.cc:288] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:33.576488 7823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:33.577360 7823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [term 1 LEADER]: Becoming Leader. State: Replica: 8d02ca9716994198899639ffc49f17df, State: Running, Role: LEADER
I20260924 12:46:33.577935 7823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02ca9716994198899639ffc49f17df" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43609 } }
I20260924 12:46:33.578532 7818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:33.580536 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d02ca9716994198899639ffc49f17df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02ca9716994198899639ffc49f17df" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43609 } } }
I20260924 12:46:33.581007 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:33.580749 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d02ca9716994198899639ffc49f17df. Latest consensus state: current_term: 1 leader_uuid: "8d02ca9716994198899639ffc49f17df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02ca9716994198899639ffc49f17df" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43609 } } }
I20260924 12:46:33.581112 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:33.581878 7831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:33.587446 7831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:33.596374 7831 catalog_manager.cc:1347] Generated new cluster ID: 438df142bb0a413fb2d1a1909725b4e0
I20260924 12:46:33.596493 7831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:33.634250 7831 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:33.636116 7831 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:33.650681 7831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d02ca9716994198899639ffc49f17df: Generated new TSK 0
I20260924 12:46:33.652518 7831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:33.674364 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:33.665986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.2.81.126:43609)
0924 12:46:33.666182 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:33.666192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:33.666512 (+ 320us) server_negotiation.cc:407] Connection header received
0924 12:46:33.666749 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:33.666753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:33.666836 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:33.666980 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0924 12:46:33.667440 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:33.668146 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:33.669763 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:33.670382 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:33.670536 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:33.671087 (+ 551us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 12:46:33.673611 (+ 2524us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 12:46:33.674057 (+ 446us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47422: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 12:46:33.674494 7821 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47422: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 12:46:33.676759 7847 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:33.665843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:43609 (local address 127.0.0.1:47422)
0924 12:46:33.666414 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:33.666430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:33.666554 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:33.667008 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:33.667013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:33.667025 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 12:46:33.667298 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:33.667307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:33.668294 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:33.668298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:33.669579 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:33.669594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:33.670624 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:33.670632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:33.670772 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:33.670812 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 12:46:33.673872 (+ 3060us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 12:46:33.676268 (+ 2396us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.81.126:43609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44777/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":1998,"thread_start_us":296,"threads_started":1}
I20260924 12:46:33.677812 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7752
I20260924 12:46:33.679155 7813 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 12:46:33.687199 7848 webserver.cc:533] Webserver: sslize failed:
E20260924 12:46:33.687325 7848 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 12:46:33.773787 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7752
2026-09-24T12:46:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (791 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T12:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:33Z Disabled control of system clock
I20260924 12:46:33.850479 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36521
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:36521
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/testchainca.cert with env {}
W20260924 12:46:34.180493 7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:34.180826 7853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:34.180919 7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:34.189373 7853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:34.189512 7853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:34.189548 7853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:34.189594 7853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:34.189654 7853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 12:46:34.189720 7853 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 12:46:34.189760 7853 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 12:46:34.200877 7853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:36521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:34.202247 7853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:34.205701 7853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:34.218837 7861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:34.218963 7858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:34.218921 7859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:34.219657 7853 server_base.cc:1034] running on GCE node
I20260924 12:46:34.220503 7853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:34.222092 7853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:34.223312 7853 hybrid_clock.cc:648] HybridClock initialized: now 1790253994223269 us; error 44 us; skew 500 ppm
I20260924 12:46:34.223658 7853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:34.226054 7853 webserver.cc:460] Webserver started at http://127.2.81.126:43255/ using document root <none> and password file <none>
I20260924 12:46:34.226773 7853 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:34.226869 7853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:34.227223 7853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:34.230333 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "1816d90064724c188d094d1f5b270d43"
format_stamp: "Formatted at 2026-09-24 12:46:34 on dist-test-slave-9tm1"
server_key: "bd6ea288288eceac0ec23929ca47d032"
server_key_iv: "db381ab4dda5c3ece6fd57695e3fc00f"
server_key_version: "encryptionkey@0"
I20260924 12:46:34.231186 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "1816d90064724c188d094d1f5b270d43"
format_stamp: "Formatted at 2026-09-24 12:46:34 on dist-test-slave-9tm1"
server_key: "bd6ea288288eceac0ec23929ca47d032"
server_key_iv: "db381ab4dda5c3ece6fd57695e3fc00f"
server_key_version: "encryptionkey@0"
I20260924 12:46:34.236469 7853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 12:46:34.240100 7867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:34.241575 7853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:34.241755 7853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "1816d90064724c188d094d1f5b270d43"
format_stamp: "Formatted at 2026-09-24 12:46:34 on dist-test-slave-9tm1"
server_key: "bd6ea288288eceac0ec23929ca47d032"
server_key_iv: "db381ab4dda5c3ece6fd57695e3fc00f"
server_key_version: "encryptionkey@0"
I20260924 12:46:34.241971 7853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:34.252789 7853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:34.253750 7853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:34.254033 7853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:34.272436 7853 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:36521
I20260924 12:46:34.272444 7918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:36521 every 8 connection(s)
I20260924 12:46:34.273922 7853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:34.277845 7919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:34.280234 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7853
I20260924 12:46:34.280440 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:34.281241 2373 external_mini_cluster.cc:1428] Setting key 974488a202a4e48624e81303e06dfa18
I20260924 12:46:34.289922 7919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43: Bootstrap starting.
I20260924 12:46:34.293109 7919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:34.294231 7919 log.cc:824] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:34.294735 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:34.283312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.2.81.126:36521)
0924 12:46:34.284485 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:34.284501 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:34.284546 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:46:34.286292 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:34.286317 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:34.286809 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:34.287206 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:34.287839 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:34.289143 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:34.290426 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:34.291122 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:34.291302 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:34.292402 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:34.292409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:34.292424 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:46:34.292951 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:34.293079 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:46:34.293104 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:34.293205 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:34.293417 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 12:46:34.293742 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":429,"threads_started":1}
I20260924 12:46:34.297273 7919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43: No bootstrap required, opened a new log
I20260924 12:46:34.298993 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:46:34.306424 7919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1816d90064724c188d094d1f5b270d43" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36521 } }
I20260924 12:46:34.306898 7919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:34.307008 7919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1816d90064724c188d094d1f5b270d43, State: Initialized, Role: FOLLOWER
I20260924 12:46:34.307652 7919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1816d90064724c188d094d1f5b270d43" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36521 } }
I20260924 12:46:34.307889 7919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:34.308022 7919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:34.308180 7919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:34.310484 7919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1816d90064724c188d094d1f5b270d43" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36521 } }
I20260924 12:46:34.311033 7919 leader_election.cc:302] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1816d90064724c188d094d1f5b270d43; no voters:
I20260924 12:46:34.311782 7919 leader_election.cc:288] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:34.311939 7924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:34.313050 7924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [term 1 LEADER]: Becoming Leader. State: Replica: 1816d90064724c188d094d1f5b270d43, State: Running, Role: LEADER
I20260924 12:46:34.313593 7924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1816d90064724c188d094d1f5b270d43" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36521 } }
I20260924 12:46:34.314036 7919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:34.316493 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1816d90064724c188d094d1f5b270d43. Latest consensus state: current_term: 1 leader_uuid: "1816d90064724c188d094d1f5b270d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1816d90064724c188d094d1f5b270d43" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36521 } } }
I20260924 12:46:34.316485 7925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1816d90064724c188d094d1f5b270d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1816d90064724c188d094d1f5b270d43" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36521 } } }
I20260924 12:46:34.317008 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:34.317013 7925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:34.319191 7932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:34.324368 7932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:34.336045 7932 catalog_manager.cc:1347] Generated new cluster ID: 6b8124705a6f4da583f458dd3f6d9d6f
I20260924 12:46:34.336197 7932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:34.346165 7932 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:34.347616 7932 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:34.361976 7932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1816d90064724c188d094d1f5b270d43: Generated new TSK 0
I20260924 12:46:34.363092 7932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:34.471696 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:34.384359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.2.81.126:36521)
0924 12:46:34.384550 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:34.384557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:34.384999 (+ 442us) server_negotiation.cc:407] Connection header received
0924 12:46:34.385215 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:34.385221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:34.385314 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:34.385484 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0924 12:46:34.385963 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:34.387052 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:34.388452 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:34.389143 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:34.389267 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:34.389792 (+ 525us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 12:46:34.470928 (+ 81136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 12:46:34.471346 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56652: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 12:46:34.471822 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:34.384191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:36521 (local address 127.0.0.1:56652)
0924 12:46:34.384874 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:34.384890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:34.385048 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:34.385517 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:34.385521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:34.385531 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 12:46:34.385822 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:34.385831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:34.387222 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:34.387226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:34.388314 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:34.388325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:34.389335 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:34.389340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:34.389462 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:34.389504 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 12:46:34.471246 (+ 81742us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 12:46:34.471485 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.81.126:36521: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":382,"threads_started":1}
W20260924 12:46:34.471840 7922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56652: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260924 12:46:34.473038 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7853
I20260924 12:46:34.474437 7914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:34.574074 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7853
2026-09-24T12:46:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (779 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (413 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1417 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:36 dist-test-slave-9tm1 krb5kdc[7964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:36 dist-test-slave-9tm1 krb5kdc[7964](info): set up 2 sockets
Sep 24 12:46:36 dist-test-slave-9tm1 krb5kdc[7964](info): commencing operation
krb5kdc: starting...
W20260924 12:46:37.425661 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:37 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:37.629529 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44599
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:44599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:37.961923 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:37.962239 7977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:37.962356 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:37.969729 7977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:37.969852 7977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:37.969902 7977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:37.969982 7977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:37.970031 7977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:37.980118 7977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:44599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:37.981488 7977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:37.982762 7977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:37.996631 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:37.996485 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:37.996582 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:37.997674 7977 server_base.cc:1034] running on GCE node
I20260924 12:46:37.998647 7977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:38.000751 7977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:38.002061 7977 hybrid_clock.cc:648] HybridClock initialized: now 1790253998002047 us; error 138 us; skew 500 ppm
I20260924 12:46:38.002485 7977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:38 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253998, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:38.010941 7977 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:46:38.013020 7977 webserver.cc:460] Webserver started at http://127.2.81.126:36237/ using document root <none> and password file <none>
I20260924 12:46:38.013815 7977 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:38.013953 7977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:38.014257 7977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:38.017506 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "22f615e8cb2a488eac7dd7efbba7eb83"
format_stamp: "Formatted at 2026-09-24 12:46:38 on dist-test-slave-9tm1"
server_key: "4439b8d9e983044258bd1ed7183e3ae0"
server_key_iv: "9680e80049883e131f5d6bf979e3a492"
server_key_version: "encryptionkey@0"
I20260924 12:46:38.018329 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "22f615e8cb2a488eac7dd7efbba7eb83"
format_stamp: "Formatted at 2026-09-24 12:46:38 on dist-test-slave-9tm1"
server_key: "4439b8d9e983044258bd1ed7183e3ae0"
server_key_iv: "9680e80049883e131f5d6bf979e3a492"
server_key_version: "encryptionkey@0"
I20260924 12:46:38.025964 7977 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 12:46:38.030140 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:38.031873 7977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:46:38.032079 7977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "22f615e8cb2a488eac7dd7efbba7eb83"
format_stamp: "Formatted at 2026-09-24 12:46:38 on dist-test-slave-9tm1"
server_key: "4439b8d9e983044258bd1ed7183e3ae0"
server_key_iv: "9680e80049883e131f5d6bf979e3a492"
server_key_version: "encryptionkey@0"
I20260924 12:46:38.032292 7977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:38.046298 7977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:38.051327 7977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:38.051697 7977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:38.070973 7977 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:44599
I20260924 12:46:38.071000 8043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:44599 every 8 connection(s)
I20260924 12:46:38.072409 7977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:38.076448 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:38.080404 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 7977
I20260924 12:46:38.080605 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:38.081277 2373 external_mini_cluster.cc:1428] Setting key 6e1392f3c3a92e68729734fd321410ca
I20260924 12:46:38.088593 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83: Bootstrap starting.
I20260924 12:46:38.092415 8044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:38.093683 8044 log.cc:824] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:46:38 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:38.097187 8044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83: No bootstrap required, opened a new log
I20260924 12:46:38.106001 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:38.083603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.2.81.126:44599)
0924 12:46:38.084480 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:38.084501 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:38.084546 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:46:38.086676 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:38.086706 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:38.087197 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:38.087564 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:38.089379 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:38.090751 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:38.092520 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:38.093314 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:38.093466 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:38.097979 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:38.097988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:38.098006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:38.098045 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:38.101358 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:38.102277 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:38.102285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:38.102314 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:38.102503 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:38.102893 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:38.102897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:38.102899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:38.103497 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:38.103737 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:38.104202 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 12:46:38.104635 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":380,"threads_started":1}
I20260924 12:46:38.106729 8044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44599 } }
I20260924 12:46:38.107112 8044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:38.107219 8044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f615e8cb2a488eac7dd7efbba7eb83, State: Initialized, Role: FOLLOWER
I20260924 12:46:38.107859 8044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44599 } }
I20260924 12:46:38.108114 8044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:38.108253 8044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:38.108417 8044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:38.110859 8044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44599 } }
I20260924 12:46:38.111568 8044 leader_election.cc:302] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f615e8cb2a488eac7dd7efbba7eb83; no voters:
I20260924 12:46:38.112228 8044 leader_election.cc:288] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:38.112388 8049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:38.113087 8049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [term 1 LEADER]: Becoming Leader. State: Replica: 22f615e8cb2a488eac7dd7efbba7eb83, State: Running, Role: LEADER
I20260924 12:46:38.113689 8049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44599 } }
I20260924 12:46:38.114406 8044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:38.116494 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44599 } } }
I20260924 12:46:38.117059 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:38.116940 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f615e8cb2a488eac7dd7efbba7eb83. Latest consensus state: current_term: 1 leader_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f615e8cb2a488eac7dd7efbba7eb83" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44599 } } }
I20260924 12:46:38.117259 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:38.119693 8057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:38.126011 8057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:38.135842 8057 catalog_manager.cc:1347] Generated new cluster ID: bb04a0dd97974750ac17a05d137502e2
I20260924 12:46:38.135982 8057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:38.147832 8057 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:38.149513 8057 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:38.157956 8057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f615e8cb2a488eac7dd7efbba7eb83: Generated new TSK 0
I20260924 12:46:38.158910 8057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:46:38.216497 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44599
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:38.544780 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:38.545142 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:38.545291 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:38.553182 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:38.553315 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:38.553463 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:38.563505 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:38.564826 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:38.566134 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:38.580467 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:38.580506 8078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:38.580439 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:38.581238 8070 server_base.cc:1034] running on GCE node
I20260924 12:46:38.581918 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:38.582965 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:38.584208 8070 hybrid_clock.cc:648] HybridClock initialized: now 1790253998584148 us; error 60 us; skew 500 ppm
I20260924 12:46:38.584556 8070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:38 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253998, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:38.587893 8070 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:46:38.592782 8070 webserver.cc:460] Webserver started at http://127.2.81.65:45097/ using document root <none> and password file <none>
I20260924 12:46:38.593897 8070 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:38.594069 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:38.594449 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:38.598829 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "cb1584e5c2af4da7afa2055efdacd1f6"
format_stamp: "Formatted at 2026-09-24 12:46:38 on dist-test-slave-9tm1"
server_key: "ddc2b7942372c4bcf42d2cf2b3192ed0"
server_key_iv: "7a055b9d0d6dc6e1d2a3011b2e65b95d"
server_key_version: "encryptionkey@0"
I20260924 12:46:38.599691 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "cb1584e5c2af4da7afa2055efdacd1f6"
format_stamp: "Formatted at 2026-09-24 12:46:38 on dist-test-slave-9tm1"
server_key: "ddc2b7942372c4bcf42d2cf2b3192ed0"
server_key_iv: "7a055b9d0d6dc6e1d2a3011b2e65b95d"
server_key_version: "encryptionkey@0"
I20260924 12:46:38.605742 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 12:46:38.609687 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:38.611351 8070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:38.611552 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "cb1584e5c2af4da7afa2055efdacd1f6"
format_stamp: "Formatted at 2026-09-24 12:46:38 on dist-test-slave-9tm1"
server_key: "ddc2b7942372c4bcf42d2cf2b3192ed0"
server_key_iv: "7a055b9d0d6dc6e1d2a3011b2e65b95d"
server_key_version: "encryptionkey@0"
I20260924 12:46:38.611902 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:38.638388 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:38.642807 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:38.643093 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:38.644125 8070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:38.645591 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:38.645673 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:38.645797 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:38.645850 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:38.678680 8070 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:43359
I20260924 12:46:38.678768 8197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:43359 every 8 connection(s)
I20260924 12:46:38.679924 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:38.686626 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8070
I20260924 12:46:38.686825 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:38.687408 2373 external_mini_cluster.cc:1428] Setting key f7e89dbe0958ee96de0706d8993304fa
Sep 24 12:46:38 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253998, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:38.700286 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:38.682418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:52021 (local address 127.2.81.126:44599)
0924 12:46:38.682869 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:38.682875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:38.683844 (+ 969us) server_negotiation.cc:407] Connection header received
0924 12:46:38.685720 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:38.685724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:38.685853 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:38.685969 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:38.687616 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:38.688373 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:38.690064 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:38.690662 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:38.690808 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:38.694278 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:38.694285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:38.694289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:38.694327 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:38.696695 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:38.697885 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:38.697891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:38.697893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:38.698032 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:38.698529 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:38.698532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:38.698534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:38.698826 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:38.698942 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:38.699833 (+ 891us) server_negotiation.cc:299] Negotiation successful
0924 12:46:38.700019 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
I20260924 12:46:38.701637 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:38.682900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44599 (local address 127.2.81.65:52021)
0924 12:46:38.683678 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:38.683717 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:38.685433 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:38.686231 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:38.686240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:38.686812 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:38.687426 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:38.687445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:38.688513 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:38.688517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:38.689912 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:38.689925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:38.690906 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:38.690912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:38.691124 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:38.691828 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:38.691851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:38.694039 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:38.696864 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:38.696874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:38.696893 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:38.697706 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:38.698159 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:38.698164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:38.698168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:38.698426 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:38.698949 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:38.698958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:38.699401 (+ 443us) client_negotiation.cc:769] Sending connection context
0924 12:46:38.699722 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 12:46:38.700261 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":312,"threads_started":1}
I20260924 12:46:38.703276 8198 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44599
I20260924 12:46:38.703596 8198 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:38.704311 8198 heartbeater.cc:511] Master 127.2.81.126:44599 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:46:38.706682 8009 ts_manager.cc:194] Registered new tserver with Master: cb1584e5c2af4da7afa2055efdacd1f6 (127.2.81.65:43359)
I20260924 12:46:38.708184 8009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:52021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:46:38.733136 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44599
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:39.070257 8204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:39.070585 8204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:39.070719 8204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:39.079908 8204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:39.080065 8204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:39.080219 8204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:39.090699 8204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:39.092341 8204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:39.094143 8204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:39.108752 8209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:39.108785 8210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:39.108966 8212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:39.109871 8204 server_base.cc:1034] running on GCE node
I20260924 12:46:39.110492 8204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:39.111529 8204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:39.112775 8204 hybrid_clock.cc:648] HybridClock initialized: now 1790253999112716 us; error 57 us; skew 500 ppm
I20260924 12:46:39.113252 8204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.116616 8204 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:46:39.118481 8204 webserver.cc:460] Webserver started at http://127.2.81.66:38223/ using document root <none> and password file <none>
I20260924 12:46:39.119299 8204 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:39.119434 8204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:39.119769 8204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:39.123015 8204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "48011c3ffdb94a8db763226b26ab33dd"
format_stamp: "Formatted at 2026-09-24 12:46:39 on dist-test-slave-9tm1"
server_key: "854afc8ce597993845629d5da4045be9"
server_key_iv: "7a679e7cbba0734c23b54e8c93df1115"
server_key_version: "encryptionkey@0"
I20260924 12:46:39.123927 8204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "48011c3ffdb94a8db763226b26ab33dd"
format_stamp: "Formatted at 2026-09-24 12:46:39 on dist-test-slave-9tm1"
server_key: "854afc8ce597993845629d5da4045be9"
server_key_iv: "7a679e7cbba0734c23b54e8c93df1115"
server_key_version: "encryptionkey@0"
I20260924 12:46:39.130897 8204 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260924 12:46:39.135188 8219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:39.137030 8204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 12:46:39.137241 8204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "48011c3ffdb94a8db763226b26ab33dd"
format_stamp: "Formatted at 2026-09-24 12:46:39 on dist-test-slave-9tm1"
server_key: "854afc8ce597993845629d5da4045be9"
server_key_iv: "7a679e7cbba0734c23b54e8c93df1115"
server_key_version: "encryptionkey@0"
I20260924 12:46:39.137491 8204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:39.157621 8204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:39.163321 8204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:39.163645 8204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:39.164708 8204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:39.166458 8204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:39.166566 8204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:39.166818 8204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:39.166899 8204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:39.199571 8204 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:37899
I20260924 12:46:39.199589 8331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:37899 every 8 connection(s)
I20260924 12:46:39.201010 8204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:39.204139 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8204
I20260924 12:46:39.204315 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:39.204720 2373 external_mini_cluster.cc:1428] Setting key af60d6a6cfbdb3126f48b7778e2e71c3
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:39.223506 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.203697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:51983 (local address 127.2.81.126:44599)
0924 12:46:39.204205 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.204211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.207062 (+ 2851us) server_negotiation.cc:407] Connection header received
0924 12:46:39.207112 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.207116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.207176 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.207255 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.209309 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.210109 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.212317 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.212786 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.212895 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.217008 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.217016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.217021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.217063 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.219682 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.220935 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.220941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.220945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.221133 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.221659 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.221662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.221665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.221930 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.222039 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.223027 (+ 988us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.223211 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
I20260924 12:46:39.225035 8335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.204265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44599 (local address 127.2.81.66:51983)
0924 12:46:39.205116 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:39.205169 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:39.206790 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:39.207559 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:39.207571 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:39.208305 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:39.209113 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:39.209140 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.210287 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.210292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:39.212136 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:39.212153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.213075 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.213083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:39.213318 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.213994 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:39.214019 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:39.216607 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:39.219988 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:39.220000 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:39.220022 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:39.220779 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:39.221252 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:39.221265 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:39.221268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:39.221547 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:39.222054 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:39.222063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:39.222587 (+ 524us) client_negotiation.cc:769] Sending connection context
0924 12:46:39.222925 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 12:46:39.223525 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":316,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:46:39.226763 8332 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44599
I20260924 12:46:39.227087 8332 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:39.227943 8332 heartbeater.cc:511] Master 127.2.81.126:44599 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:46:39.229784 8009 ts_manager.cc:194] Registered new tserver with Master: 48011c3ffdb94a8db763226b26ab33dd (127.2.81.66:37899)
I20260924 12:46:39.230835 8009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:51983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:46:39.257458 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44599
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:39.599566 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:39.599859 8338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:39.599992 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:39.607565 8338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:39.607704 8338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:39.607867 8338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:39.618860 8338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:39.620219 8338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:39.621729 8338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:39.635456 8343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:39.635404 8344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:39.635592 8346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:39.636281 8338 server_base.cc:1034] running on GCE node
I20260924 12:46:39.637441 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:39.638568 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:39.639817 8338 hybrid_clock.cc:648] HybridClock initialized: now 1790253999639764 us; error 46 us; skew 500 ppm
I20260924 12:46:39.640188 8338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.643929 8338 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:46:39.645970 8338 webserver.cc:460] Webserver started at http://127.2.81.67:34763/ using document root <none> and password file <none>
I20260924 12:46:39.646812 8338 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:39.646948 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:39.647267 8338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:39.650377 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "dd4b4e3a0aac4439b267e0f91be85525"
format_stamp: "Formatted at 2026-09-24 12:46:39 on dist-test-slave-9tm1"
server_key: "6eecbe746a730f2888c6404f671a3343"
server_key_iv: "c5e3bd31e3c77a764f7e2f8df2c93b3f"
server_key_version: "encryptionkey@0"
I20260924 12:46:39.651197 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "dd4b4e3a0aac4439b267e0f91be85525"
format_stamp: "Formatted at 2026-09-24 12:46:39 on dist-test-slave-9tm1"
server_key: "6eecbe746a730f2888c6404f671a3343"
server_key_iv: "c5e3bd31e3c77a764f7e2f8df2c93b3f"
server_key_version: "encryptionkey@0"
I20260924 12:46:39.656909 8338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:46:39.661165 8353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:39.662781 8338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:46:39.662986 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "dd4b4e3a0aac4439b267e0f91be85525"
format_stamp: "Formatted at 2026-09-24 12:46:39 on dist-test-slave-9tm1"
server_key: "6eecbe746a730f2888c6404f671a3343"
server_key_iv: "c5e3bd31e3c77a764f7e2f8df2c93b3f"
server_key_version: "encryptionkey@0"
I20260924 12:46:39.663203 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:39.683503 8338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:39.687853 8338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:39.688130 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:39.689141 8338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:39.690517 8338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:39.690600 8338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:39.690740 8338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:39.690804 8338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:39.711490 8198 heartbeater.cc:503] Master 127.2.81.126:44599 was elected leader, sending a full tablet report...
I20260924 12:46:39.722877 8338 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:34479
I20260924 12:46:39.722911 8465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:34479 every 8 connection(s)
I20260924 12:46:39.724190 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:39.728713 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8338
I20260924 12:46:39.728889 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:39.729444 2373 external_mini_cluster.cc:1428] Setting key 44c6945e40592502a2ec6a654d301969
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:46:39.748901 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.726856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:37165 (local address 127.2.81.126:44599)
0924 12:46:39.727321 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.727327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.728249 (+ 922us) server_negotiation.cc:407] Connection header received
0924 12:46:39.730509 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.730513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.730598 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.730695 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.732888 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.733717 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.736435 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.736951 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.737111 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.740863 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.740869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.740874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.740913 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.744015 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.745170 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.745176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.745181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.745372 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.746124 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.746129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.746132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.746823 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.747047 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.748175 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.748504 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":197,"threads_started":1}
I20260924 12:46:39.750031 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.727392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44599 (local address 127.2.81.67:37165)
0924 12:46:39.728092 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:39.728130 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:39.730240 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:39.730958 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:39.730967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:39.731568 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:39.732620 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:39.732649 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.733911 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.733916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:39.736179 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:39.736228 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.737236 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.737242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:39.737496 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.738273 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:39.738331 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:39.740578 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:39.744238 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:39.744249 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:39.744270 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:39.745023 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:39.745525 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:39.745531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:39.745535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:39.745967 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:39.747040 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:39.747049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:39.747466 (+ 417us) client_negotiation.cc:769] Sending connection context
0924 12:46:39.747864 (+ 398us) client_negotiation.cc:240] Negotiation successful
0924 12:46:39.748724 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":267,"threads_started":1}
I20260924 12:46:39.752068 8466 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44599
I20260924 12:46:39.752408 8466 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:39.753242 8466 heartbeater.cc:511] Master 127.2.81.126:44599 requested a full tablet report, sending...
I20260924 12:46:39.755056 8009 ts_manager.cc:194] Registered new tserver with Master: dd4b4e3a0aac4439b267e0f91be85525 (127.2.81.67:34479)
I20260924 12:46:39.755927 8009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:37165
I20260924 12:46:39.759933 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/test-admin.keytab.
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.792053 2373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 12:46:39.797413 8474 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:39.794862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.2.81.65:43359)
0924 12:46:39.795538 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.795545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.795570 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:46:39.795633 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.795639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.795870 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.796031 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.796964 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45588: BlockingRecv error: recv got EOF from 127.0.0.1:45588 (error 108)
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":372,"threads_started":1}
W20260924 12:46:39.797503 2373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc' of type FILE
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.799540 2373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.801031 2373 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:39.816464 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.802836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.2.81.65:43359)
0924 12:46:39.803033 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.803039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.803158 (+ 119us) server_negotiation.cc:407] Connection header received
0924 12:46:39.803317 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.803321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.803379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.803459 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.804477 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.805505 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.806777 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.807355 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.807483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.810844 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.810856 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.810872 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.810918 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.813654 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.814348 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.814356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.814358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.814489 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.814935 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.814939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.814941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.815326 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.815532 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.815892 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.816133 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 12:46:39.822412 8474 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:39.820627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.2.81.65:43359)
0924 12:46:39.820797 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.820803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.821163 (+ 360us) server_negotiation.cc:407] Connection header received
0924 12:46:39.821328 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.821333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.821394 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.821471 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.822178 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45612: BlockingRecv error: recv got EOF from 127.0.0.1:45612 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 12:46:39.822590 2373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc' of type FILE
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.824115 2373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.825433 2373 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:39.839972 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.826838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.2.81.65:43359)
0924 12:46:39.827031 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.827037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.827208 (+ 171us) server_negotiation.cc:407] Connection header received
0924 12:46:39.827389 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.827394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.827457 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.827545 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.828487 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.829325 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.830534 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.830953 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.831048 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.834460 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.834466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.834469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.834504 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.837229 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.837967 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.837973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.837975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.838109 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.838535 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.838539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.838541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.838886 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.839086 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.839456 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.839698 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 12:46:39.844507 8474 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:39.842732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.2.81.65:43359)
0924 12:46:39.842929 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.842935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.843154 (+ 219us) server_negotiation.cc:407] Connection header received
0924 12:46:39.843351 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.843355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.843427 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.843507 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.844295 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45638: BlockingRecv error: recv got EOF from 127.0.0.1:45638 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 12:46:39.844642 2373 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790253949676453-2373-0/krb5kdc/krb5cc' of type FILE
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.846285 2373 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:39.847517 2373 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:46:39.864305 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.848845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.2.81.65:43359)
0924 12:46:39.849076 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.849083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.849413 (+ 330us) server_negotiation.cc:407] Connection header received
0924 12:46:39.849640 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.849648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.849753 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.849924 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.851182 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.851945 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.853942 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.854409 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.854516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.858927 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.858933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.858936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.858977 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.861537 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.862268 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.862272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.862274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.862428 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.862892 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.862895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.862897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.863159 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.863454 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.863772 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.863954 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:46:39.865851 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8070
I20260924 12:46:39.878371 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.866581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.2.81.65:43359)
0924 12:46:39.866812 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.866818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.867080 (+ 262us) server_negotiation.cc:407] Connection header received
0924 12:46:39.867218 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.867222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.867299 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.867388 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.868563 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.869351 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.870740 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.871169 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.871268 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.873342 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.873347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.873351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.873387 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.875856 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.876522 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.876527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.876529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.876664 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.876962 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.876966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.876968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.877261 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.877466 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.877830 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.878049 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:46:39.879019 8193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:39.961556 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8070
I20260924 12:46:39.973979 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8204
Sep 24 12:46:39 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:46:39.990339 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:39.974748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.2.81.66:37899)
0924 12:46:39.975210 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:39.975217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:39.975240 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:46:39.975328 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:39.975336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:39.975633 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:39.975789 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:39.976994 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.978022 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.979432 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:39.980239 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:39.980388 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:39.984584 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:39.984601 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:39.984623 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:39.984671 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:39.987468 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.988207 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.988214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.988217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.988345 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:39.988694 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:39.988698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:39.988700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:39.989178 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:39.989461 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:39.989797 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 12:46:39.990042 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260924 12:46:39.991510 8327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:40.073235 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8204
I20260924 12:46:40.087904 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8338
Sep 24 12:46:40 dist-test-slave-9tm1 krb5kdc[7964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790253999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:46:40.104916 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:40.088887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.2.81.67:34479)
0924 12:46:40.089530 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:40.089536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:40.089572 (+ 36us) server_negotiation.cc:407] Connection header received
0924 12:46:40.089634 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:40.089641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:40.089890 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:40.090028 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:40.091118 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:40.092207 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:40.093648 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:40.094251 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:40.094362 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:40.098702 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:40.098720 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:40.098745 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:40.098795 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:40.101746 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:40.102668 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:40.102678 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:40.102681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:40.102836 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:40.103308 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:40.103312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:40.103314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:40.103715 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:40.104018 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:40.104364 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 12:46:40.104582 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260924 12:46:40.106426 8461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:40.190544 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8338
I20260924 12:46:40.202569 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 7977
I20260924 12:46:40.203771 8039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:40.303860 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 7977
2026-09-24T12:46:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3902 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:40 dist-test-slave-9tm1 krb5kdc[8494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:40 dist-test-slave-9tm1 krb5kdc[8494](info): set up 2 sockets
Sep 24 12:46:40 dist-test-slave-9tm1 krb5kdc[8494](info): commencing operation
krb5kdc: starting...
W20260924 12:46:41.325798 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:41 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab.
Entry for principal oryx/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab.
I20260924 12:46:41.533574 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33615
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab
--principal=oryx/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:33615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:41.872283 8507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:41.872592 8507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:41.872690 8507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:41.879925 8507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:41.880043 8507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:41.880087 8507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:41.880142 8507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:41.880184 8507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:41.890446 8507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:33615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab
--principal=oryx/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:41.891734 8507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:41.893021 8507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:41.904294 8513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:41.904212 8515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:41.904321 8512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:41.905047 8507 server_base.cc:1034] running on GCE node
I20260924 12:46:41.905831 8507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:41.907289 8507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:41.908528 8507 hybrid_clock.cc:648] HybridClock initialized: now 1790254001908471 us; error 51 us; skew 500 ppm
I20260924 12:46:41.908869 8507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:41 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:41.912087 8507 init.cc:377] Logged in from keytab as oryx/127.2.81.126@KRBTEST.COM (short username oryx)
I20260924 12:46:41.913654 8507 webserver.cc:460] Webserver started at http://127.2.81.126:45167/ using document root <none> and password file <none>
I20260924 12:46:41.914324 8507 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:41.914438 8507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:41.914713 8507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:41.917732 8507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "b1a8c06366a8492b997e0e594acf767f"
format_stamp: "Formatted at 2026-09-24 12:46:41 on dist-test-slave-9tm1"
server_key: "d6ecdc5899a4e173aa9542efbf036434"
server_key_iv: "ff6630e026f49247baab30c2fd38037f"
server_key_version: "encryptionkey@0"
I20260924 12:46:41.918475 8507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "b1a8c06366a8492b997e0e594acf767f"
format_stamp: "Formatted at 2026-09-24 12:46:41 on dist-test-slave-9tm1"
server_key: "d6ecdc5899a4e173aa9542efbf036434"
server_key_iv: "ff6630e026f49247baab30c2fd38037f"
server_key_version: "encryptionkey@0"
I20260924 12:46:41.923887 8507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:46:41.927348 8522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:41.928711 8507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 12:46:41.928876 8507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "b1a8c06366a8492b997e0e594acf767f"
format_stamp: "Formatted at 2026-09-24 12:46:41 on dist-test-slave-9tm1"
server_key: "d6ecdc5899a4e173aa9542efbf036434"
server_key_iv: "ff6630e026f49247baab30c2fd38037f"
server_key_version: "encryptionkey@0"
I20260924 12:46:41.929105 8507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:41.942967 8507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:41.948167 8507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:41.948470 8507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:41.967993 8507 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:33615
I20260924 12:46:41.968019 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:33615 every 8 connection(s)
I20260924 12:46:41.969427 8507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:41.973264 8574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:41.974159 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8507
I20260924 12:46:41.974363 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:41.975169 2373 external_mini_cluster.cc:1428] Setting key fcc6f672b38ecb5980bf68c595294e1e
I20260924 12:46:41.987638 8574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Bootstrap starting.
Sep 24 12:46:41 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:41.991485 8574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:41.992666 8574 log.cc:824] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:41.996083 8574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: No bootstrap required, opened a new log
I20260924 12:46:41.999148 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:41.977351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.2.81.126:33615)
0924 12:46:41.978114 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:41.978130 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:41.978170 (+ 40us) server_negotiation.cc:407] Connection header received
0924 12:46:41.979767 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:41.979791 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:41.980253 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:41.980683 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:41.982468 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:41.983778 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:41.985575 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:41.986331 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:41.986480 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:41.991209 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:41.991220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:41.991237 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:41.991281 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:41.994715 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:41.995618 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:41.995624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:41.995631 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:41.995813 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:41.996325 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:41.996330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:41.996332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:41.996917 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:41.997227 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:41.997648 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 12:46:41.998046 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":318,"threads_started":1}
I20260924 12:46:42.004307 8574 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } }
I20260924 12:46:42.004702 8574 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:42.004802 8574 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a8c06366a8492b997e0e594acf767f, State: Initialized, Role: FOLLOWER
I20260924 12:46:42.005393 8574 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } }
I20260924 12:46:42.005656 8574 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:42.005795 8574 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:42.005988 8574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:42.008273 8574 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } }
I20260924 12:46:42.008819 8574 leader_election.cc:302] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1a8c06366a8492b997e0e594acf767f; no voters:
I20260924 12:46:42.009460 8574 leader_election.cc:288] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:42.009593 8579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:42.010687 8579 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 LEADER]: Becoming Leader. State: Replica: b1a8c06366a8492b997e0e594acf767f, State: Running, Role: LEADER
I20260924 12:46:42.011301 8579 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } }
I20260924 12:46:42.011675 8574 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:42.014075 8581 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a8c06366a8492b997e0e594acf767f. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } }
I20260924 12:46:42.014605 8581 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:42.015411 8580 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } }
I20260924 12:46:42.015741 8580 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:42.017979 8588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:42.022646 8588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:42.031786 8588 catalog_manager.cc:1347] Generated new cluster ID: bdf4338ba97f47d087a3c81e17d8f646
I20260924 12:46:42.031913 8588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:42.078612 8588 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:42.080511 8588 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:42.095924 8588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Generated new TSK 0
I20260924 12:46:42.097110 8588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab.
Entry for principal oryx/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab.
I20260924 12:46:42.152949 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33615
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:42.476909 8600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:42.477226 8600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:42.477346 8600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:42.484668 8600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:42.484786 8600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:42.484937 8600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:42.494999 8600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:42.496331 8600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:42.497736 8600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:42.511039 8606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:42.511058 8605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:42.510993 8608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:42.511953 8600 server_base.cc:1034] running on GCE node
I20260924 12:46:42.512553 8600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:42.513587 8600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:42.514794 8600 hybrid_clock.cc:648] HybridClock initialized: now 1790254002514739 us; error 44 us; skew 500 ppm
I20260924 12:46:42.515133 8600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:42 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254002, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:42.518507 8600 init.cc:377] Logged in from keytab as oryx/127.2.81.65@KRBTEST.COM (short username oryx)
I20260924 12:46:42.520057 8600 webserver.cc:460] Webserver started at http://127.2.81.65:45545/ using document root <none> and password file <none>
I20260924 12:46:42.520748 8600 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:42.520870 8600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:42.521209 8600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:42.524248 8600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "080d3e378290483b8ea3aa4691656e67"
format_stamp: "Formatted at 2026-09-24 12:46:42 on dist-test-slave-9tm1"
server_key: "ad7753cff4a674fb33163e7f95ef5b9a"
server_key_iv: "4223fe6bbb632a74b56566cd6d9cc1fa"
server_key_version: "encryptionkey@0"
I20260924 12:46:42.525086 8600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "080d3e378290483b8ea3aa4691656e67"
format_stamp: "Formatted at 2026-09-24 12:46:42 on dist-test-slave-9tm1"
server_key: "ad7753cff4a674fb33163e7f95ef5b9a"
server_key_iv: "4223fe6bbb632a74b56566cd6d9cc1fa"
server_key_version: "encryptionkey@0"
I20260924 12:46:42.530351 8600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 12:46:42.533927 8615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:42.535328 8600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:46:42.535490 8600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "080d3e378290483b8ea3aa4691656e67"
format_stamp: "Formatted at 2026-09-24 12:46:42 on dist-test-slave-9tm1"
server_key: "ad7753cff4a674fb33163e7f95ef5b9a"
server_key_iv: "4223fe6bbb632a74b56566cd6d9cc1fa"
server_key_version: "encryptionkey@0"
I20260924 12:46:42.535688 8600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:42.551506 8600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:42.555920 8600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:42.556226 8600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:42.557278 8600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:42.558700 8600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:42.558782 8600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:42.558913 8600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:42.558971 8600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:42.589733 8600 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:34053
I20260924 12:46:42.589773 8727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:34053 every 8 connection(s)
I20260924 12:46:42.591457 8600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:42.593539 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8600
I20260924 12:46:42.593753 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:42.594681 2373 external_mini_cluster.cc:1428] Setting key 875d79e5de8c5ed1193c1455bfc571b0
WARNING: no policy specified for oryx/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.66@KRBTEST.COM" created.
Sep 24 12:46:42 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254002, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:42.630625 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:42.595762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:56519 (local address 127.2.81.126:33615)
0924 12:46:42.604687 (+ 8925us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:42.604694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:42.615178 (+ 10484us) server_negotiation.cc:407] Connection header received
0924 12:46:42.615259 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:42.615264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:42.615375 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:42.615487 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:42.617178 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:42.618323 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:42.620150 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:42.620622 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:42.620726 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:42.624857 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:42.624863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:42.624866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:42.624901 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:42.627447 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:42.628190 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:42.628196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:42.628200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:42.628334 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:42.628766 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:42.628770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:42.628773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:42.629138 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:42.629282 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:42.630076 (+ 794us) server_negotiation.cc:299] Negotiation successful
0924 12:46:42.630304 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8787,"thread_start_us":10387,"threads_started":1}
I20260924 12:46:42.631533 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:42.595009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:56519)
0924 12:46:42.612183 (+ 17174us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:42.612267 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:42.614726 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:42.615807 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:42.615818 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:42.616352 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:42.616948 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:42.616989 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:42.618540 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:42.618545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:42.619972 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:42.619988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:42.620876 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:42.620884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:42.621179 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:42.621858 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:42.621887 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:42.624544 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:42.627603 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:42.627614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:42.627632 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:42.628060 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:42.628445 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:42.628449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:42.628451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:42.628651 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:42.629287 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:42.629294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:42.629642 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 12:46:42.629901 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 12:46:42.630294 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16831,"thread_start_us":16593,"threads_started":1}
I20260924 12:46:42.633242 8728 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33615
I20260924 12:46:42.633574 8728 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:42.634467 8728 heartbeater.cc:511] Master 127.2.81.126:33615 requested a full tablet report, sending...
I20260924 12:46:42.637107 8539 ts_manager.cc:194] Registered new tserver with Master: 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053)
I20260924 12:46:42.639150 8539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:56519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab.
Entry for principal oryx/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab.
I20260924 12:46:42.649271 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33615
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:42.979538 8734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:42.980475 8734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:42.981253 8734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:42.990422 8734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:42.990586 8734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:42.990747 8734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:43.001744 8734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:43.003557 8734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:43.005686 8734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:43.020797 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:43.020824 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:43.021003 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:43.021410 8734 server_base.cc:1034] running on GCE node
I20260924 12:46:43.022027 8734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:43.023074 8734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:43.024302 8734 hybrid_clock.cc:648] HybridClock initialized: now 1790254003024252 us; error 47 us; skew 500 ppm
I20260924 12:46:43.024644 8734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:43 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254003, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:43.032598 8734 init.cc:377] Logged in from keytab as oryx/127.2.81.66@KRBTEST.COM (short username oryx)
I20260924 12:46:43.034397 8734 webserver.cc:460] Webserver started at http://127.2.81.66:36709/ using document root <none> and password file <none>
I20260924 12:46:43.035161 8734 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:43.035283 8734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:43.035581 8734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:43.038676 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "2a26fdf02f374e7c93839150131ea8a5"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "20389b6f1bd8187720536b80c8e8412e"
server_key_iv: "7c5f3a6479617a46a3075197fbd8fa6c"
server_key_version: "encryptionkey@0"
I20260924 12:46:43.039494 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "2a26fdf02f374e7c93839150131ea8a5"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "20389b6f1bd8187720536b80c8e8412e"
server_key_iv: "7c5f3a6479617a46a3075197fbd8fa6c"
server_key_version: "encryptionkey@0"
I20260924 12:46:43.044950 8734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:46:43.048822 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:43.050341 8734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 12:46:43.050508 8734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "2a26fdf02f374e7c93839150131ea8a5"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "20389b6f1bd8187720536b80c8e8412e"
server_key_iv: "7c5f3a6479617a46a3075197fbd8fa6c"
server_key_version: "encryptionkey@0"
I20260924 12:46:43.050710 8734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:43.075096 8734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:43.079752 8734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:43.080142 8734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:43.081348 8734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:43.083144 8734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:43.083250 8734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:43.083369 8734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:43.083431 8734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:43.115559 8734 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:45181
I20260924 12:46:43.115638 8861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:45181 every 8 connection(s)
I20260924 12:46:43.116830 8734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:43.120018 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8734
I20260924 12:46:43.120173 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:43.120779 2373 external_mini_cluster.cc:1428] Setting key 0a12b14531f2325d0a7941aae2c26b04
Sep 24 12:46:43 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254003, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:43.137946 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:43.119480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:48197 (local address 127.2.81.126:33615)
0924 12:46:43.119695 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:43.119709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:43.121031 (+ 1322us) server_negotiation.cc:407] Connection header received
0924 12:46:43.123218 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:43.123222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:43.123290 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:43.123380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:43.125423 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.126162 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.127761 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.128174 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.128285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:43.132060 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:43.132065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:43.132069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:43.132105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:43.134754 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:43.135591 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:43.135595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:43.135597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:43.135730 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:43.136161 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:43.136165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:43.136167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:43.136429 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:43.136556 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:43.137363 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 12:46:43.137615 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 12:46:43.138993 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:43.119981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:48197)
0924 12:46:43.120815 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:43.120886 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:43.122881 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:43.123658 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:43.123669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:43.124383 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:43.125207 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:43.125239 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.126342 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.126346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:43.127613 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:43.127627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.128402 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.128409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:43.128624 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:43.129317 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:43.129340 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:43.131806 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:43.134949 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:43.134959 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:43.134975 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:43.135454 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:43.135846 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:43.135851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:43.135855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:43.136048 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:43.136553 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:43.136562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:43.136946 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 12:46:43.137248 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 12:46:43.137819 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":321,"threads_started":1}
I20260924 12:46:43.140544 8862 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33615
I20260924 12:46:43.140895 8862 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:46:43.141803 8862 heartbeater.cc:511] Master 127.2.81.126:33615 requested a full tablet report, sending...
I20260924 12:46:43.143196 8539 ts_manager.cc:194] Registered new tserver with Master: 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181)
I20260924 12:46:43.144016 8539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:48197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab.
Entry for principal oryx/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab.
I20260924 12:46:43.168886 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33615
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:43.498476 8867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:43.498807 8867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:43.498937 8867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:43.506433 8867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:43.506575 8867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:43.506729 8867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:43.516719 8867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:43.518039 8867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:43.519332 8867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:43.532809 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:43.532909 8872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:43.532930 8873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:43.533519 8867 server_base.cc:1034] running on GCE node
I20260924 12:46:43.534163 8867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:43.535161 8867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:43.536387 8867 hybrid_clock.cc:648] HybridClock initialized: now 1790254003536334 us; error 47 us; skew 500 ppm
I20260924 12:46:43.536725 8867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:43 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254003, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:43.540381 8867 init.cc:377] Logged in from keytab as oryx/127.2.81.67@KRBTEST.COM (short username oryx)
I20260924 12:46:43.541980 8867 webserver.cc:460] Webserver started at http://127.2.81.67:36973/ using document root <none> and password file <none>
I20260924 12:46:43.542716 8867 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:43.542840 8867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:43.543128 8867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:43.546259 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "218b188c1ed545c990b93f8fae3a7258"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "b584d54a0d4157d8467beb044d9f6d95"
server_key_iv: "a0430114a8fc5a80cb9ddb16765a663a"
server_key_version: "encryptionkey@0"
I20260924 12:46:43.547096 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "218b188c1ed545c990b93f8fae3a7258"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "b584d54a0d4157d8467beb044d9f6d95"
server_key_iv: "a0430114a8fc5a80cb9ddb16765a663a"
server_key_version: "encryptionkey@0"
I20260924 12:46:43.552963 8867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:46:43.557132 8882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:43.558841 8867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:43.559022 8867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "218b188c1ed545c990b93f8fae3a7258"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "b584d54a0d4157d8467beb044d9f6d95"
server_key_iv: "a0430114a8fc5a80cb9ddb16765a663a"
server_key_version: "encryptionkey@0"
I20260924 12:46:43.559243 8867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:43.576493 8867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:43.581386 8867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:43.581669 8867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:43.582643 8867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:43.584098 8867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:43.584178 8867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:43.584285 8867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:43.584335 8867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:43.616096 8867 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:42451
I20260924 12:46:43.616153 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:42451 every 8 connection(s)
I20260924 12:46:43.617401 8867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:43.619309 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 8867
I20260924 12:46:43.619477 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:43.619899 2373 external_mini_cluster.cc:1428] Setting key 9faeff60276b7df26c51c12e67b547bf
Sep 24 12:46:43 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254003, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:43.639148 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:43.620365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:56163 (local address 127.2.81.126:33615)
0924 12:46:43.620569 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:43.620575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:43.622117 (+ 1542us) server_negotiation.cc:407] Connection header received
0924 12:46:43.624048 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:43.624052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:43.624136 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:43.624225 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:43.625729 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.626460 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.628049 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.628473 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.628583 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:43.632385 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:43.632390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:43.632393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:43.632428 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:43.635698 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:43.636565 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:43.636570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:43.636572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:43.636723 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:43.637261 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:43.637265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:43.637268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:43.637567 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:43.637689 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:43.638600 (+ 911us) server_negotiation.cc:299] Negotiation successful
0924 12:46:43.638839 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:46:43.640142 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:43.621091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:56163)
0924 12:46:43.621968 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:43.622020 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:43.623792 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:43.624467 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:43.624476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:43.625002 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:43.625560 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:43.625578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.626617 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.626621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:43.627904 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:43.627918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:43.628705 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:43.628712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:43.628944 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:43.629638 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:43.629671 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:43.632128 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:43.635884 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:43.635895 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:43.635915 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:43.636416 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:43.636851 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:43.636855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:43.636857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:43.637121 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:43.637722 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:43.637729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:43.638243 (+ 514us) client_negotiation.cc:769] Sending connection context
0924 12:46:43.638495 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 12:46:43.638946 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":315,"threads_started":1}
I20260924 12:46:43.641594 8995 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33615
I20260924 12:46:43.641906 8995 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:43.642413 8728 heartbeater.cc:503] Master 127.2.81.126:33615 was elected leader, sending a full tablet report...
I20260924 12:46:43.642660 8995 heartbeater.cc:511] Master 127.2.81.126:33615 requested a full tablet report, sending...
I20260924 12:46:43.644089 8539 ts_manager.cc:194] Registered new tserver with Master: 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451)
I20260924 12:46:43.644860 8539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:56163
I20260924 12:46:43.648361 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.81.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.125.keytab.
Entry for principal oryx/127.2.81.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.125.keytab.
I20260924 12:46:43.697532 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:43707
--webserver_interface=127.2.81.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.125.keytab
--principal=oryx/127.2.81.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:33615,127.2.81.125:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:44.027241 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:44.027556 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:44.027688 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:44.035104 9000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:44.035212 9000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:44.035266 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:44.035338 9000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:44.035403 9000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:44.045924 9000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:33615,127.2.81.125:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.125.keytab
--principal=oryx/127.2.81.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.81.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
W20260924 12:46:44.046305 9000 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.81.126:33615,127.2.81.125:43707'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 12:46:44.047358 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:44.048672 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:44.060523 9006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:44.060577 9008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:44.060552 9005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:44.061247 9000 server_base.cc:1034] running on GCE node
I20260924 12:46:44.062044 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:44.063458 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:44.064705 9000 hybrid_clock.cc:648] HybridClock initialized: now 1790254004064642 us; error 34 us; skew 500 ppm
I20260924 12:46:44.065124 9000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:44.068557 9000 init.cc:377] Logged in from keytab as oryx/127.2.81.125@KRBTEST.COM (short username oryx)
I20260924 12:46:44.070269 9000 webserver.cc:460] Webserver started at http://127.2.81.125:45795/ using document root <none> and password file <none>
I20260924 12:46:44.071009 9000 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:44.071117 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:44.071432 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:44.074620 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/instance:
uuid: "c6955d68527f48e0b77b0903ad0b08ec"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "47f281366883b1d89121191b3b7ae17f"
server_key_iv: "eb3e8168f9dcc4c05450fe24e9abde14"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.075453 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal/instance:
uuid: "c6955d68527f48e0b77b0903ad0b08ec"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "47f281366883b1d89121191b3b7ae17f"
server_key_iv: "eb3e8168f9dcc4c05450fe24e9abde14"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.081032 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:46:44.085080 9015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:44.086609 9000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 12:46:44.086829 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
uuid: "c6955d68527f48e0b77b0903ad0b08ec"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "47f281366883b1d89121191b3b7ae17f"
server_key_iv: "eb3e8168f9dcc4c05450fe24e9abde14"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.087059 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:44.118206 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:44.122781 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:44.123031 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:44.140390 9000 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.125:43707
I20260924 12:46:44.140467 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.125:43707 every 8 connection(s)
I20260924 12:46:44.141788 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
I20260924 12:46:44.145725 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:44.146729 8862 heartbeater.cc:503] Master 127.2.81.126:33615 was elected leader, sending a full tablet report...
I20260924 12:46:44.147655 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 9000
I20260924 12:46:44.155596 9067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.125@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:44.170873 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.149901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.2.81.125:43707)
0924 12:46:44.150705 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.150717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.150758 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:46:44.152338 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.152362 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.152784 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.153189 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.154948 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.156346 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.157613 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.158389 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.158583 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.163657 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.163668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.163681 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.163718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.166718 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.167507 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.167514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.167521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.167668 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.168099 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.168103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.168105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.168727 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.169050 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.169498 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.169882 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":353,"threads_started":1}
I20260924 12:46:44.172935 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.158104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:45294)
0924 12:46:44.159071 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.159116 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.159282 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.159651 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.159656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.160265 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.160499 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.160507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.161954 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.161960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.163580 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.163593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.164286 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.164291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.164460 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.165094 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.165110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.167630 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.170570 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.170581 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.170588 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.171027 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.171374 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.171379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.171383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.171517 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.172049 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.172056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.172234 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.172398 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.172632 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":629,"threads_started":1}
I20260924 12:46:44.173100 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.158108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.2.81.126:33615)
0924 12:46:44.158581 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.158586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.159215 (+ 629us) server_negotiation.cc:407] Connection header received
0924 12:46:44.159407 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.159411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.159472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.159559 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.160662 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.161807 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.163715 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.164117 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.164240 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.167821 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.167826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.167829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.167863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.170442 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.171129 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.171134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.171138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.171266 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.171612 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.171617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.171620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.171919 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.172057 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.172519 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.172751 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260924 12:46:44.174499 9067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.125@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
I20260924 12:46:44.186806 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.175086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:58010)
0924 12:46:44.175282 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.175301 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.175407 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.175649 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.175653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.176008 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.176132 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.176140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.177069 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.177074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.177935 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.177945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.178594 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.178600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.178769 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.179331 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.179348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.181623 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.184541 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.184546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.184551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.184916 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.185291 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.185296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.185300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.185415 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.185905 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.185909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.186044 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.186157 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.186443 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 12:46:44.186878 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.175206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.2.81.125:43707)
0924 12:46:44.175330 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.175335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.175351 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:46:44.175500 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.175505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.175553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.175625 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.176253 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.176929 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.178059 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.178459 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.178565 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.181928 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.181933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.181937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.181969 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.184393 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.185076 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.185081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.185085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.185197 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.185508 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.185513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.185516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.185801 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.185922 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.186243 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.186582 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":43}
I20260924 12:46:44.191854 9067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Bootstrap starting.
I20260924 12:46:44.194962 9067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:44.195945 9067 log.cc:824] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:44.198522 9067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: No bootstrap required, opened a new log
I20260924 12:46:44.205546 9067 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } }
I20260924 12:46:44.206076 9067 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:44.206185 9067 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6955d68527f48e0b77b0903ad0b08ec, State: Initialized, Role: FOLLOWER
I20260924 12:46:44.206753 9067 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } }
I20260924 12:46:44.207613 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } } }
I20260924 12:46:44.207937 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:46:44.208580 9067 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: configured and running, proceeding with master startup.
W20260924 12:46:44.217365 9085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:46:44.217530 9085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:46:44.219762 9000 master_runner.cc:428] Detected that this master c6955d68527f48e0b77b0903ad0b08ec is joining an existing cluster
I20260924 12:46:44.219877 9000 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.81.125:43707
I20260924 12:46:44.241155 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.224309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:45306)
0924 12:46:44.225690 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.225713 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.225905 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.226459 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.226463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.227230 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.227697 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.227713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.229409 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.229416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.231225 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.231245 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.232677 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.232684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.232879 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.233614 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.233636 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.234475 (+ 839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.238324 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.238329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.238334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.238773 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.239183 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.239188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.239195 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.239353 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.240039 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.240047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.240231 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.240394 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.240691 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":989,"threads_started":1}
I20260924 12:46:44.241998 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.224287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.2.81.126:33615)
0924 12:46:44.224477 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.224482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.225789 (+ 1307us) server_negotiation.cc:407] Connection header received
0924 12:46:44.226068 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.226074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.226158 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.226234 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.227936 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.229172 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.231451 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.232178 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.232283 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.234686 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.234692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.234696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.234737 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.238169 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.238916 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.238921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.238924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.239062 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.239475 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.239480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.239482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.239844 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.239986 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.241479 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.241695 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.2.81.124@KRBTEST.COM; defaulting to no policy
I20260924 12:46:44.252554 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.225885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:58024)
0924 12:46:44.234869 (+ 8984us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.234895 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.235020 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.235384 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.235388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.236019 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.236331 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.236353 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.237826 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.237832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.239387 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.239401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.240836 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.240844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.241067 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.241713 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.241746 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.242804 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.247685 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.247690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.247694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.248250 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.250225 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.250231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.250236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.250419 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.251325 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.251330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.251489 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.251659 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.252009 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8833,"thread_start_us":7617,"threads_started":1}
I20260924 12:46:44.253386 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.226319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.2.81.125:43707)
0924 12:46:44.226519 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.226525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.234987 (+ 8462us) server_negotiation.cc:407] Connection header received
0924 12:46:44.235131 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.235137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.235213 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.235326 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.236529 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.237635 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.239745 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.240431 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.240525 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.242993 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.242999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.243004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.243038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.247479 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.248400 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.248406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.248410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.250108 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.250544 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.250549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.250553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.251074 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.251250 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.252912 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.253135 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.124@KRBTEST.COM" created.
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.126@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
I20260924 12:46:44.271420 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.249053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.2.81.125:43707)
0924 12:46:44.253002 (+ 3949us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.253009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.253031 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:46:44.253126 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.253131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.253222 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.253378 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.255758 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.257157 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.258927 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.259652 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.259804 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.264141 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.264148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.264154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.264195 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.267839 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.268927 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.268932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.268936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.269139 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.269617 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.269622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.269625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.270053 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.270184 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.270767 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.270991 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3758,"thread_start_us":3450,"threads_started":1}
I20260924 12:46:44.271433 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.249013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:58036)
0924 12:46:44.249867 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.249907 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.250112 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.253681 (+ 3569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.253715 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.254812 (+ 1097us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.255178 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.255195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.257337 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.257343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.258731 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.258751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.259883 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.259889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.260095 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.260893 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.260914 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.263868 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.268049 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.268068 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.268077 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.268771 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.269280 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.269286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.269290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.269494 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.270206 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.270216 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.270433 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.270654 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.271015 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":406,"threads_started":1}
I20260924 12:46:44.273742 8539 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.81.125:43707: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec"
member_type: NON_VOTER
last_known_addr {
host: "127.2.81.125"
port: 43707
}
attrs {
promote: true
}
}
dest_uuid: "b1a8c06366a8492b997e0e594acf767f"
cas_config_opid_index: -1
I20260924 12:46:44.274971 8539 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } }
I20260924 12:46:44.278481 9099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c6955d68527f48e0b77b0903ad0b08ec (127.2.81.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } }
I20260924 12:46:44.280547 9042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:44.282500 9099 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.81.125:43707
I20260924 12:46:44.282961 9099 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } }
I20260924 12:46:44.283172 9099 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:44.283383 9000 master.cc:537] Master@127.2.81.125:43707 shutting down...
I20260924 12:46:44.284852 9042 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 12:46:44.286340 9099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.124.keytab.
Entry for principal oryx/127.2.81.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.124.keytab.
W20260924 12:46:44.290197 8523 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 12:46:44.290805 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:43453
--webserver_interface=127.2.81.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.124.keytab
--principal=oryx/127.2.81.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:44.294245 8523 consensus_peers.cc:487] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f -> Peer c6955d68527f48e0b77b0903ad0b08ec (127.2.81.125:43707): Couldn't send request to peer c6955d68527f48e0b77b0903ad0b08ec. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 12:46:44.295828 9000 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:44.296216 9000 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:44.296406 9000 tablet_replica.cc:330] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: stopping tablet replica
I20260924 12:46:44.303165 9000 master.cc:552] Master@127.2.81.125:43707 shutdown complete.
I20260924 12:46:44.303273 9000 master_runner.cc:305] Clearing existing system tablet
I20260924 12:46:44.303694 9000 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:44.303792 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:44.306170 9000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 12:46:44.307636 9104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:44.308069 9000 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:44.308305 9000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 12:46:44.308418 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
uuid: "c6955d68527f48e0b77b0903ad0b08ec"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "47f281366883b1d89121191b3b7ae17f"
server_key_iv: "eb3e8168f9dcc4c05450fe24e9abde14"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.312482 9000 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:44.317736 9000 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 12:46:44.318017 9000 log.cc:1196] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 12:46:44.318495 9000 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Deleting consensus metadata
I20260924 12:46:44.318742 9000 master_runner.cc:315] Copying system tablet from 127.2.81.126:33615
I20260924 12:46:44.327199 9000 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: tablet copy: Beginning tablet copy session from remote peer at address 127.2.81.126:33615
I20260924 12:46:44.340581 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.328317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:45308)
0924 12:46:44.328923 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.328942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.329158 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.329526 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.329531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.329954 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.330152 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.330162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.331393 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.331398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.332372 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.332387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.333311 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.333317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.333491 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.334125 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.334143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.334860 (+ 717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.337850 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.337856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.337861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.338278 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.338671 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.338675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.338678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.338861 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.339392 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.339397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.339577 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.339712 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.340122 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":379,"threads_started":1}
I20260924 12:46:44.340602 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.328374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.2.81.126:33615)
0924 12:46:44.328558 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.328564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.329053 (+ 489us) server_negotiation.cc:407] Connection header received
0924 12:46:44.329321 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.329325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.329391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.329497 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.330302 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.331204 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.332518 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.333158 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.333283 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.335074 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.335079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.335083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.335116 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.337688 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.338411 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.338415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.338417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.338547 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.338962 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.338967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.338971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.339265 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.339402 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.340050 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.340271 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 12:46:44.341681 8559 tablet_copy_service.cc:140] P b1a8c06366a8492b997e0e594acf767f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c6955d68527f48e0b77b0903ad0b08ec ({username='oryx', principal='oryx/127.2.81.125@KRBTEST.COM'} at 127.0.0.1:45308)
I20260924 12:46:44.341917 8559 tablet_copy_service.cc:161] P b1a8c06366a8492b997e0e594acf767f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c6955d68527f48e0b77b0903ad0b08ec at {username='oryx', principal='oryx/127.2.81.125@KRBTEST.COM'} at 127.0.0.1:45308: session id = c6955d68527f48e0b77b0903ad0b08ec-00000000000000000000000000000000
I20260924 12:46:44.345052 8559 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 12:46:44.347070 9000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 12:46:44.352195 9000 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: tablet copy: Starting download of 0 data blocks...
I20260924 12:46:44.352525 9000 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: tablet copy: Starting download of 1 WAL segments...
I20260924 12:46:44.355027 9000 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 12:46:44.358927 8559 tablet_copy_service.cc:342] P b1a8c06366a8492b997e0e594acf767f: Request end of tablet copy session c6955d68527f48e0b77b0903ad0b08ec-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.81.125@KRBTEST.COM'} at 127.0.0.1:45308
I20260924 12:46:44.359261 8559 tablet_copy_service.cc:433] P b1a8c06366a8492b997e0e594acf767f: ending tablet copy session c6955d68527f48e0b77b0903ad0b08ec-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c6955d68527f48e0b77b0903ad0b08ec
I20260924 12:46:44.369565 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:44.372067 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:44.381403 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:44.382721 9000 server_base.cc:1034] running on GCE node
W20260924 12:46:44.383482 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:44.383940 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:44.384559 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:44.385772 9000 hybrid_clock.cc:648] HybridClock initialized: now 1790254004385707 us; error 36 us; skew 500 ppm
I20260924 12:46:44.386158 9000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:44.387938 9000 init.cc:377] Logged in from keytab as oryx/127.2.81.125@KRBTEST.COM (short username oryx)
I20260924 12:46:44.389102 9000 webserver.cc:460] Webserver started at http://127.2.81.125:43175/ using document root <none> and password file <none>
I20260924 12:46:44.389405 9000 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:44.389475 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:44.391830 9000 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 12:46:44.393379 9121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:44.393752 9000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 12:46:44.393875 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
uuid: "c6955d68527f48e0b77b0903ad0b08ec"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "47f281366883b1d89121191b3b7ae17f"
server_key_iv: "eb3e8168f9dcc4c05450fe24e9abde14"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.394018 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:44.403525 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:44.407024 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:44.436932 9000 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.125:43707
I20260924 12:46:44.436933 9172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.125:43707 every 8 connection(s)
I20260924 12:46:44.438023 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
I20260924 12:46:44.443373 9173 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:46:44.447187 9173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Bootstrap starting.
I20260924 12:46:44.466130 9173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:46:44.466797 9173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Bootstrap complete.
I20260924 12:46:44.467875 9173 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } }
I20260924 12:46:44.468158 9173 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6955d68527f48e0b77b0903ad0b08ec, State: Initialized, Role: LEARNER
I20260924 12:46:44.468410 9173 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } }
I20260924 12:46:44.469448 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } }
I20260924 12:46:44.469739 9173 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:44.469893 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: LEARNER
I20260924 12:46:44.497545 9187 catalog_manager.cc:1259] Loaded cluster ID: bdf4338ba97f47d087a3c81e17d8f646
I20260924 12:46:44.497694 9187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: loading cluster ID for follower catalog manager: success
I20260924 12:46:44.501605 9187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: acquiring CA information for follower catalog manager: success
I20260924 12:46:44.505132 9187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 12:46:44.647883 8995 heartbeater.cc:503] Master 127.2.81.126:33615 was elected leader, sending a full tablet report...
I20260924 12:46:44.690620 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.681800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:38276)
0924 12:46:44.682083 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.682102 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.682220 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.683204 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.683210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.683246 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:44.683486 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.683500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.685581 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.685587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.687644 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.687672 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.689723 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.689729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.689953 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.690019 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.690133 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.690300 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260924 12:46:44.690671 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.682114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.2.81.125:43707)
0924 12:46:44.682788 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.682795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.682817 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:46:44.682895 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.682900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.682996 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.683117 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:44.683680 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.685385 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.687896 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.689481 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.689638 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.689887 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.690182 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.690324 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":372,"threads_started":1}
I20260924 12:46:44.692414 9148 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 12:46:44.693110 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a8c06366a8492b997e0e594acf767f. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } }
I20260924 12:46:44.693382 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: LEARNER
W20260924 12:46:44.719055 9102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:44.719411 9102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:44.719501 9102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:44.727573 9102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:44.727700 9102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:44.727748 9102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:44.727811 9102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:44.727855 9102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:44.739070 9102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.124.keytab
--principal=oryx/127.2.81.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:43453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.81.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:44.740650 9102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:44.742242 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:44.755497 9196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:44.755470 9193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:44.756486 9194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:44.756600 9102 server_base.cc:1034] running on GCE node
I20260924 12:46:44.757536 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:44.759042 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:44.760277 9102 hybrid_clock.cc:648] HybridClock initialized: now 1790254004760238 us; error 59 us; skew 500 ppm
I20260924 12:46:44.760656 9102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:44.764629 9102 init.cc:377] Logged in from keytab as oryx/127.2.81.124@KRBTEST.COM (short username oryx)
I20260924 12:46:44.766316 9102 webserver.cc:460] Webserver started at http://127.2.81.124:46561/ using document root <none> and password file <none>
I20260924 12:46:44.767094 9102 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:44.767215 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:44.767503 9102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:44.770640 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/instance:
uuid: "552350fb926a4eedb5ac2690443d8625"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "afdcf109d09b484cf16a3c2c1814637b"
server_key_iv: "97be7be49b0e75c59e4f14aa4a43003c"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.771427 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal/instance:
uuid: "552350fb926a4eedb5ac2690443d8625"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "afdcf109d09b484cf16a3c2c1814637b"
server_key_iv: "97be7be49b0e75c59e4f14aa4a43003c"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.777415 9102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 12:46:44.781080 9203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:44.782481 9102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:46:44.782645 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
uuid: "552350fb926a4eedb5ac2690443d8625"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "afdcf109d09b484cf16a3c2c1814637b"
server_key_iv: "97be7be49b0e75c59e4f14aa4a43003c"
server_key_version: "encryptionkey@0"
I20260924 12:46:44.782874 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:44.817512 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:44.822415 9102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:44.822726 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:44.842566 9102 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.124:43453
I20260924 12:46:44.842597 9254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.124:43453 every 8 connection(s)
I20260924 12:46:44.843909 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
I20260924 12:46:44.847656 9255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:44.848711 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 9102
I20260924 12:46:44.857318 9255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.124@KRBTEST.COM
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.124@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:44.874454 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.850877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.2.81.124:43453)
0924 12:46:44.851674 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.851687 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.851770 (+ 83us) server_negotiation.cc:407] Connection header received
0924 12:46:44.854359 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.854388 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.854866 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.855276 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.856953 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.858496 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.859776 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.860902 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.861129 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.866233 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.866245 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.866287 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.866351 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.869750 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.870530 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.870539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.870550 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.870746 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.871142 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.871176 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.871179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.871762 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.872037 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.872521 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.873083 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":338,"threads_started":1}
I20260924 12:46:44.875047 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.860005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.2.81.126:33615)
0924 12:46:44.861190 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.861196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.861213 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:46:44.861267 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.861273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.861345 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.861455 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.862643 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.863400 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.864867 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.865385 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.865492 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.869539 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.869547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.869552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.869596 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.872232 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.873011 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.873017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.873021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.873170 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.873561 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.873566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.873569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.873893 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.874027 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.874540 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.874754 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":975,"threads_started":1}
I20260924 12:46:44.876091 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.859865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37306)
0924 12:46:44.860572 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.860613 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.860775 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.861501 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.861506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.862252 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.862482 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.862490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.863533 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.863537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.864670 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.864685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.865628 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.865648 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.865857 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.866598 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.866620 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.869306 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.872380 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.872391 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.872399 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.872842 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.873303 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.873307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.873309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.873454 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.874063 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.874069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.874215 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.874404 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.875747 (+ 1343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":285,"threads_started":1}
I20260924 12:46:44.877810 9255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.124@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
I20260924 12:46:44.891001 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.878688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.2.81.125:43707)
0924 12:46:44.878925 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.878929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.878944 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:46:44.878990 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.878994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.879048 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.879126 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.879828 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.880522 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.881721 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.882119 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.882225 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.885538 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.885543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.885546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.885581 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.888404 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.889134 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.889138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.889140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.889276 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.889655 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.889659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.889661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.889945 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.890079 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.890436 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.890657 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:46:44.890991 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.878468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:38290)
0924 12:46:44.878673 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.878689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.878792 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.879166 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.879170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.879538 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.879676 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.879684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.880638 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.880643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.881591 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.881602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.882261 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.882267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.882486 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.883243 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.883261 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.885325 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.888559 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.888566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.888570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.889012 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.889393 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.889399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.889403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.889541 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.890059 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.890063 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.890202 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.890314 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.890641 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 12:46:44.892313 9255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254004, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.124@KRBTEST.COM for oryx/127.2.81.124@KRBTEST.COM
I20260924 12:46:44.904733 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.893238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.0.0.1:60668)
0924 12:46:44.893422 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.893437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.893532 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.893778 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.893781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.894119 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.894302 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.894311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.895315 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.895319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.896193 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.896205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.896832 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.896836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.897027 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.897564 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.897580 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.899542 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.902246 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.902250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.902253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.902604 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.902921 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.902924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.902926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.903076 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.903962 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.903965 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.904111 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.904239 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.904445 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 12:46:44.904798 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.893271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.2.81.124:43453)
0924 12:46:44.893434 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.893438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.893480 (+ 42us) server_negotiation.cc:407] Connection header received
0924 12:46:44.893606 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.893609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.893661 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.893743 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.894447 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.895149 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.896317 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.896705 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.896798 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.899730 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.899735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.899738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.899765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.902089 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.902699 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.902702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.902705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.902806 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.903160 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.903163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.903165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.903855 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.903964 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.904327 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.904563 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":53}
I20260924 12:46:44.909937 9255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Bootstrap starting.
I20260924 12:46:44.913472 9255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:44.914598 9255 log.cc:824] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:44.917470 9255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: No bootstrap required, opened a new log
I20260924 12:46:44.924777 9255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } }
I20260924 12:46:44.925174 9255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:44.925263 9255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552350fb926a4eedb5ac2690443d8625, State: Initialized, Role: FOLLOWER
I20260924 12:46:44.926348 9255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } }
I20260924 12:46:44.927280 9262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } } }
I20260924 12:46:44.927593 9262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:46:44.928267 9255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 12:46:44.937553 9273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:46:44.937682 9273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:46:44.941915 9102 master_runner.cc:428] Detected that this master 552350fb926a4eedb5ac2690443d8625 is joining an existing cluster
I20260924 12:46:44.942072 9102 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.81.124:43453
I20260924 12:46:44.965298 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.946558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.2.81.126:33615)
0924 12:46:44.946740 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.946747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.949583 (+ 2836us) server_negotiation.cc:407] Connection header received
0924 12:46:44.949860 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.949866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.949949 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.950108 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.951746 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.952858 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.955155 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.955832 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.955931 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.958576 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.958583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.958588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.958632 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.962091 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.962964 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.962971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.962974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.963122 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.963557 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.963563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.963566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.963906 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.964023 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.964688 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.964910 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:46:44.966063 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.946809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.2.81.124:43453)
0924 12:46:44.947005 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.947011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.947997 (+ 986us) server_negotiation.cc:407] Connection header received
0924 12:46:44.948307 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.948313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.948394 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.948484 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.949934 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.951292 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.953293 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.953987 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.954084 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.959648 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.959654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.959658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.959690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.963250 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.963989 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.963993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.963996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.964135 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.964515 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.964519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.964522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.964870 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.965027 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.965496 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.965700 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:46:44.967156 9280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.948357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37316)
0924 12:46:44.949470 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.949488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.949698 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.950437 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.950442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.951146 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.951548 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.951558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.953089 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.953094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.954985 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.954997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.956269 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.956276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.956440 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.957112 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.957134 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.958348 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.962263 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.962268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.962272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.962816 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.963270 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.963275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.963278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.963435 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.964095 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.964100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.964381 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.965850 (+ 1469us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.966523 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"spinlock_wait_cycles":50176,"thread_start_us":779,"threads_started":1}
I20260924 12:46:44.967661 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.946827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.0.0.1:60678)
0924 12:46:44.947463 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.947483 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.948056 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.948691 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.948695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.949428 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.949806 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.949817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.951539 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.951544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.953133 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.953148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.954902 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.954911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.955073 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.956228 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.956255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.959485 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.963431 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.963435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.963439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.963877 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.964253 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.964257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.964261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.964416 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.965086 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.965091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.965248 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.966266 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.966728 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":53,"thread_start_us":369,"threads_started":1}
I20260924 12:46:44.978221 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.950121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:38302)
0924 12:46:44.957785 (+ 7664us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.957805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.957942 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.958323 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.958328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.958945 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.959285 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.959301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.960712 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.960717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.962486 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.962501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.964344 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.964352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.964527 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.965172 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.965196 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.966273 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.975097 (+ 8824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.975104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.975109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.975644 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.976136 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.976141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.976144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.976321 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.977051 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.977055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.977295 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.977442 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.977767 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7487,"spinlock_wait_cycles":896,"thread_start_us":7327,"threads_started":1}
I20260924 12:46:44.978233 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.949819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.2.81.125:43707)
0924 12:46:44.950000 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.950006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.957921 (+ 7915us) server_negotiation.cc:407] Connection header received
0924 12:46:44.958107 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.958113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.958199 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.958295 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.959448 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.960541 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.962667 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.963865 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.963970 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.966470 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.966477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.966481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.966523 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.974915 (+ 8392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.975801 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.975807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.975811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.975984 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.976452 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.976457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.976461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.976816 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.976942 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.977604 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.977858 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:46:44.981470 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8600
Sep 24 12:46:44 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.126@KRBTEST.COM for oryx/127.2.81.124@KRBTEST.COM
I20260924 12:46:44.995235 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.976293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.0.0.1:60686)
0924 12:46:44.976711 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:44.976732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:44.976840 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:44.977608 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:44.977612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:44.978094 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:44.978290 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.978303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.980158 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.980164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.981477 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:44.981493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.982458 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.982464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:44.982628 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.983354 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:44.983376 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:44.987193 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:44.992021 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.992027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.992031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.992553 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.992960 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:44.992965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:44.992991 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:44.993132 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:44.993944 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:44.993949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:44.994116 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 12:46:44.994248 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 12:46:44.994852 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260924 12:46:44.996312 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.977086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.2.81.124:43453)
0924 12:46:44.977249 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:44.977255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:44.977270 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:46:44.977368 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:44.977372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:44.977453 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:44.978446 (+ 993us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:44.978590 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.979551 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.981667 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:44.982263 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:44.982973 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:44.987542 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:44.987550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:44.987555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:44.987614 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:44.991825 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.992699 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.992704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.992707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.992840 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:44.993322 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:44.993329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:44.993332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:44.993760 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:44.993889 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:44.995728 (+ 1839us) server_negotiation.cc:299] Negotiation successful
0924 12:46:44.995973 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 12:46:44.998337 8539 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.81.124:43453: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "552350fb926a4eedb5ac2690443d8625"
member_type: NON_VOTER
last_known_addr {
host: "127.2.81.124"
port: 43453
}
attrs {
promote: true
}
}
dest_uuid: "b1a8c06366a8492b997e0e594acf767f"
cas_config_opid_index: 5
I20260924 12:46:44.999418 8539 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } }
I20260924 12:46:45.001298 9099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 552350fb926a4eedb5ac2690443d8625 (127.2.81.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.001971 9148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 LEARNER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 12:46:45.004681 9284 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 12:46:45.005421 9230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:45.009160 9099 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.81.124:43453
I20260924 12:46:45.012231 9283 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.012748 9283 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.013522 9148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 552350fb926a4eedb5ac2690443d8625 (127.2.81.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.017057 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: NON_VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: true } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.017354 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: LEARNER
I20260924 12:46:45.017840 9099 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: attempting to promote NON_VOTER c6955d68527f48e0b77b0903ad0b08ec to VOTER
I20260924 12:46:45.018569 9099 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } }
I20260924 12:46:45.020596 9148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 LEARNER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 12:46:45.021178 9283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 12:46:45.025736 9148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6955d68527f48e0b77b0903ad0b08ec (127.2.81.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.65@KRBTEST.COM
I20260924 12:46:45.029249 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.029476 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:46:45.025030 9287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6955d68527f48e0b77b0903ad0b08ec (127.2.81.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.033358 9099 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.033635 9099 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.033932 9287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.034209 9287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.037262 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:44.982514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.2.81.65:34053)
0924 12:46:45.012163 (+ 29649us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.012174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.012257 (+ 83us) server_negotiation.cc:407] Connection header received
0924 12:46:45.012339 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.012348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.012663 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.014948 (+ 2285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:45.015215 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.016863 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.019348 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.022445 (+ 3097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.023576 (+ 1131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.028749 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:45.028768 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:45.028791 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:45.028847 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:45.033209 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.034237 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.034248 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.034252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.034424 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.034802 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.034807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.034811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.035270 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:45.035562 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.035960 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.036253 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29462,"thread_start_us":30320,"threads_started":1}
I20260924 12:46:45.037593 9102 master.cc:537] Master@127.2.81.124:43453 shutting down...
I20260924 12:46:45.037689 9230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 12:46:45.038861 8723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:45.039224 9229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 12:46:45.041769 9287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:46:45.048828 8525 consensus_peers.cc:487] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f -> Peer 552350fb926a4eedb5ac2690443d8625 (127.2.81.124:43453): Couldn't send request to peer 552350fb926a4eedb5ac2690443d8625. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 12:46:45.050429 9102 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:45.050835 9102 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:45.051029 9102 tablet_replica.cc:330] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: stopping tablet replica
I20260924 12:46:45.069136 9102 master.cc:552] Master@127.2.81.124:43453 shutdown complete.
I20260924 12:46:45.069260 9102 master_runner.cc:305] Clearing existing system tablet
I20260924 12:46:45.069697 9102 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:45.069824 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:45.072836 9102 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 12:46:45.074586 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:45.074991 9102 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:45.075270 9102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 12:46:45.075371 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
uuid: "552350fb926a4eedb5ac2690443d8625"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "afdcf109d09b484cf16a3c2c1814637b"
server_key_iv: "97be7be49b0e75c59e4f14aa4a43003c"
server_key_version: "encryptionkey@0"
I20260924 12:46:45.080411 9102 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:45.088507 9102 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 12:46:45.088696 9102 log.cc:1196] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 12:46:45.089624 9102 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Deleting consensus metadata
I20260924 12:46:45.089882 9102 master_runner.cc:315] Copying system tablet from 127.2.81.126:33615
I20260924 12:46:45.092475 9102 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: tablet copy: Beginning tablet copy session from remote peer at address 127.2.81.126:33615
I20260924 12:46:45.116376 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.093328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37324)
0924 12:46:45.101519 (+ 8191us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:45.101552 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:45.101741 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:45.102149 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:45.102158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:45.102706 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:45.102972 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.102987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.104460 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.104466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.106014 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.106031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.107008 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.107015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.107188 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.108047 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:45.108072 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:45.109185 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:45.113338 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.113344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.113349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.113879 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.114315 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.114320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.114324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.114520 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.115205 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:45.115210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:45.115389 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 12:46:45.115576 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 12:46:45.115922 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8013,"thread_start_us":7825,"threads_started":1}
I20260924 12:46:45.117475 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.093482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.2.81.126:33615)
0924 12:46:45.093705 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.093711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.101701 (+ 7990us) server_negotiation.cc:407] Connection header received
0924 12:46:45.101888 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.101895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.101983 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.102100 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:45.103169 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.104295 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.106176 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.106835 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.106968 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.109475 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:45.109483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:45.109488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:45.109531 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:45.113172 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.114020 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.114027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.114031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.114202 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.114632 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.114637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.114641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.114984 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:45.115124 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.116854 (+ 1730us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.117118 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 12:46:45.119073 8559 tablet_copy_service.cc:140] P b1a8c06366a8492b997e0e594acf767f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 552350fb926a4eedb5ac2690443d8625 ({username='oryx', principal='oryx/127.2.81.124@KRBTEST.COM'} at 127.0.0.1:37324)
I20260924 12:46:45.119256 8559 tablet_copy_service.cc:161] P b1a8c06366a8492b997e0e594acf767f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 552350fb926a4eedb5ac2690443d8625 at {username='oryx', principal='oryx/127.2.81.124@KRBTEST.COM'} at 127.0.0.1:37324: session id = 552350fb926a4eedb5ac2690443d8625-00000000000000000000000000000000
I20260924 12:46:45.123034 8559 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 12:46:45.125207 9102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 12:46:45.133054 9102 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: tablet copy: Starting download of 0 data blocks...
I20260924 12:46:45.133414 9102 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: tablet copy: Starting download of 1 WAL segments...
I20260924 12:46:45.135998 9102 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 12:46:45.140770 8559 tablet_copy_service.cc:342] P b1a8c06366a8492b997e0e594acf767f: Request end of tablet copy session 552350fb926a4eedb5ac2690443d8625-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.81.124@KRBTEST.COM'} at 127.0.0.1:37324
I20260924 12:46:45.140945 8559 tablet_copy_service.cc:433] P b1a8c06366a8492b997e0e594acf767f: ending tablet copy session 552350fb926a4eedb5ac2690443d8625-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 552350fb926a4eedb5ac2690443d8625
I20260924 12:46:45.154469 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 12:46:45.154534 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8600
W20260924 12:46:45.158068 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:45.164458 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:45.168458 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:45.168771 9102 server_base.cc:1034] running on GCE node
I20260924 12:46:45.169274 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:45.169976 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:45.171408 9102 hybrid_clock.cc:648] HybridClock initialized: now 1790254005171335 us; error 60 us; skew 500 ppm
I20260924 12:46:45.171689 9102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:46:45.172230 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:34053
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:33615
--webserver_port=45545
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254005, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:45.173868 9102 init.cc:377] Logged in from keytab as oryx/127.2.81.124@KRBTEST.COM (short username oryx)
I20260924 12:46:45.180109 9102 webserver.cc:460] Webserver started at http://127.2.81.124:36997/ using document root <none> and password file <none>
I20260924 12:46:45.180450 9102 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:45.180543 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:45.189186 9102 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260924 12:46:45.192046 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:45.192535 9102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 12:46:45.192679 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
uuid: "552350fb926a4eedb5ac2690443d8625"
format_stamp: "Formatted at 2026-09-24 12:46:44 on dist-test-slave-9tm1"
server_key: "afdcf109d09b484cf16a3c2c1814637b"
server_key_iv: "97be7be49b0e75c59e4f14aa4a43003c"
server_key_version: "encryptionkey@0"
I20260924 12:46:45.192881 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:45.228430 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:45.232867 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:45.264269 9102 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.124:43453
I20260924 12:46:45.264344 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.124:43453 every 8 connection(s)
I20260924 12:46:45.265424 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
I20260924 12:46:45.271579 9361 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:46:45.275653 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Bootstrap starting.
I20260924 12:46:45.296306 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:46:45.296851 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: Bootstrap complete.
I20260924 12:46:45.297969 9361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } }
I20260924 12:46:45.298285 9361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 552350fb926a4eedb5ac2690443d8625, State: Initialized, Role: LEARNER
I20260924 12:46:45.298573 9361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } }
I20260924 12:46:45.299424 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.299820 9361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:45.299948 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: This master's current role is: LEARNER
I20260924 12:46:45.307219 9375 catalog_manager.cc:1259] Loaded cluster ID: bdf4338ba97f47d087a3c81e17d8f646
I20260924 12:46:45.307314 9375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: loading cluster ID for follower catalog manager: success
I20260924 12:46:45.309944 9375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: acquiring CA information for follower catalog manager: success
I20260924 12:46:45.311877 9375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 12:46:45.553069 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.546015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.2.81.124:43453)
0924 12:46:45.546620 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.546625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.546893 (+ 268us) server_negotiation.cc:407] Connection header received
0924 12:46:45.547086 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.547091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.547177 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.547386 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:45.548061 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.548960 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.551259 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.552130 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.552256 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.552429 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.552685 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.552792 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":327,"threads_started":1}
I20260924 12:46:45.553126 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.545812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.0.0.1:60694)
0924 12:46:45.546243 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:45.546262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:45.546414 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:45.547437 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:45.547442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:45.547475 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:45.547789 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.547809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.549149 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.549154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.551054 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.551076 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.552338 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.552345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.552520 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.552558 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:46:45.552663 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 12:46:45.552810 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":216,"threads_started":1}
I20260924 12:46:45.554355 9336 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260924 12:46:45.554996 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a8c06366a8492b997e0e594acf767f. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.555274 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: This master's current role is: LEARNER
W20260924 12:46:45.577857 9307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:45.578153 9307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:45.578256 9307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:45.586472 9307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:45.586596 9307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:45.586709 9307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:45.596824 9307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=45545
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:45.598133 9307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:45.599387 9307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:45.613535 9383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:45.613560 9382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:45.613726 9385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:45.614326 9307 server_base.cc:1034] running on GCE node
I20260924 12:46:45.614975 9307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:45.616096 9307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:45.617316 9307 hybrid_clock.cc:648] HybridClock initialized: now 1790254005617265 us; error 50 us; skew 500 ppm
I20260924 12:46:45.617664 9307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254005, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:45.621119 9307 init.cc:377] Logged in from keytab as oryx/127.2.81.65@KRBTEST.COM (short username oryx)
I20260924 12:46:45.622623 9307 webserver.cc:460] Webserver started at http://127.2.81.65:45545/ using document root <none> and password file <none>
I20260924 12:46:45.623484 9307 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:45.623641 9307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:45.629739 9307 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:46:45.632548 9392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:45.634153 9307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:46:45.634343 9307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "080d3e378290483b8ea3aa4691656e67"
format_stamp: "Formatted at 2026-09-24 12:46:42 on dist-test-slave-9tm1"
server_key: "ad7753cff4a674fb33163e7f95ef5b9a"
server_key_iv: "4223fe6bbb632a74b56566cd6d9cc1fa"
server_key_version: "encryptionkey@0"
I20260924 12:46:45.635344 9307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:45.675621 9307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:45.680235 9307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:45.680965 9307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:45.682085 9307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:45.683471 9307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:45.683573 9307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:45.683681 9307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:45.683748 9307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:45.721427 9307 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:34053
I20260924 12:46:45.721511 9504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:34053 every 8 connection(s)
I20260924 12:46:45.722826 9307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:45.730811 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 9307
I20260924 12:46:45.731173 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8734
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254005, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.124@KRBTEST.COM
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254005, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254005, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
Sep 24 12:46:45 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.66@KRBTEST.COM
I20260924 12:46:45.752228 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.725710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:50399 (local address 127.2.81.124:43453)
0924 12:46:45.725909 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.725916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.728149 (+ 2233us) server_negotiation.cc:407] Connection header received
0924 12:46:45.730663 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.730670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.730768 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.730841 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:45.734515 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.735602 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.738402 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.738892 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.739022 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.744280 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:45.744287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:45.744292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:45.744334 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:45.747720 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.749287 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.749293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.749298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.749433 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.749897 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.749903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.749906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.750287 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:45.750433 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.751606 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.751906 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:46:45.752246 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.725597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:45225 (local address 127.2.81.126:33615)
0924 12:46:45.726789 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.726795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.728147 (+ 1352us) server_negotiation.cc:407] Connection header received
0924 12:46:45.731063 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.731068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.731144 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.731217 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:45.734500 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.735629 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.738401 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.739021 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.739138 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.744288 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:45.744296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:45.744301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:45.744345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:45.748192 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.749877 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.749883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.749886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.750048 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.750501 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.750507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.750510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.750830 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:45.750970 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.751599 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.751861 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":979,"threads_started":1}
I20260924 12:46:45.753731 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.728792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:40639 (local address 127.2.81.125:43707)
0924 12:46:45.729766 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.729770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.729789 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:46:45.730824 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.730829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.730927 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.731028 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:45.734514 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.735635 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.740470 (+ 4835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.741208 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.741331 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.745085 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:45.745092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:45.745096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:45.745149 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:45.748695 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.749912 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.749918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.749921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.750057 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.750873 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.750884 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.750887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.751283 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:45.751488 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.753118 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.753387 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":796,"threads_started":1}
I20260924 12:46:45.754118 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.726364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.2.81.65:50399)
0924 12:46:45.727966 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:45.728017 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:45.730324 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:45.731290 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:45.731302 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:45.733170 (+ 1868us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:45.734277 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.734306 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.735835 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.735841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.738163 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.738230 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.739138 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.739146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.739377 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.740132 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:45.740167 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:45.743979 (+ 3812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:45.747930 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.747941 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.747962 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.748631 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.749548 (+ 917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.749553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.749557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.749771 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.750504 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:45.750513 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:45.751163 (+ 650us) client_negotiation.cc:769] Sending connection context
0924 12:46:45.752501 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0924 12:46:45.753048 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":322,"thread_start_us":962,"threads_started":1}
I20260924 12:46:45.754153 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.726365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:45225)
0924 12:46:45.727966 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:45.728022 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:45.730324 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:45.731455 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:45.731460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:45.733130 (+ 1670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:45.734277 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.734306 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.735836 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.735842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.738218 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.738233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.739430 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.739436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.739768 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.740262 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:45.740286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:45.743979 (+ 3693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:45.748768 (+ 4789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.748773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.748777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.749735 (+ 958us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.750195 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.750200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.750203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.750368 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.751045 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:45.751049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:45.751185 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 12:46:45.752508 (+ 1323us) client_negotiation.cc:240] Negotiation successful
0924 12:46:45.752822 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":470,"spinlock_wait_cycles":63488,"thread_start_us":871,"threads_started":1}
I20260924 12:46:45.754505 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.728652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.2.81.65:40639)
0924 12:46:45.729409 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:45.729423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:45.730350 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:45.731291 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:45.731302 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:45.733698 (+ 2396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:45.734277 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.734306 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.735835 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.735842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.740262 (+ 4420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:45.740278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.741576 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.741582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:45.741766 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.742377 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:45.742400 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:45.744804 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:45.748862 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.748867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.748870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.749425 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.750516 (+ 1091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:45.750520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:45.750523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:45.750694 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:45.751551 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:45.751556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:45.751702 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 12:46:45.751862 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 12:46:45.752419 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":1880,"thread_start_us":519,"threads_started":1}
I20260924 12:46:45.756119 9505 heartbeater.cc:347] Connected to a master server at 127.2.81.124:43453
I20260924 12:46:45.756175 9507 heartbeater.cc:347] Connected to a master server at 127.2.81.125:43707
I20260924 12:46:45.756088 9506 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33615
I20260924 12:46:45.757107 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:45.732107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.2.81.66:45181)
0924 12:46:45.732667 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:45.732677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:45.733249 (+ 572us) server_negotiation.cc:407] Connection header received
0924 12:46:45.733363 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:45.733372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:45.733652 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:45.733950 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:45.736924 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.738459 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.741361 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:45.742221 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:45.742340 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:45.748287 (+ 5947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:45.748307 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:45.748328 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:45.748373 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:45.752013 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.753433 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.753442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.753446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.753613 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:45.754868 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:45.754892 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:45.754895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:45.755341 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:45.755643 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:45.756216 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 12:46:45.756506 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":260,"threads_started":1}
I20260924 12:46:45.757120 9506 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:45.757131 9507 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:45.757141 9505 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:45.758303 9505 heartbeater.cc:511] Master 127.2.81.124:43453 requested a full tablet report, sending...
I20260924 12:46:45.758323 9507 heartbeater.cc:511] Master 127.2.81.125:43707 requested a full tablet report, sending...
I20260924 12:46:45.758366 8857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:45.758843 9506 heartbeater.cc:511] Master 127.2.81.126:33615 requested a full tablet report, sending...
I20260924 12:46:45.760649 8539 ts_manager.cc:194] Re-registered known tserver with Master: 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053)
I20260924 12:46:45.761104 9138 ts_manager.cc:194] Registered new tserver with Master: 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053)
I20260924 12:46:45.761766 8539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:45225
I20260924 12:46:45.762270 9326 ts_manager.cc:194] Registered new tserver with Master: 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053)
I20260924 12:46:45.845263 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8734
I20260924 12:46:45.858093 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:45181
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:33615
--webserver_port=36709
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:46:45.983033 9287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: NON_VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: true } } }
I20260924 12:46:45.983384 9287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.984877 9518 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: attempting to promote NON_VOTER 552350fb926a4eedb5ac2690443d8625 to VOTER
I20260924 12:46:45.985666 9518 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:45.987876 9148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 12:46:45.988231 9336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 LEARNER]: Refusing update from remote peer b1a8c06366a8492b997e0e594acf767f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 12:46:45.988463 9287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 12:46:45.988847 9521 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 12:46:45.991463 9518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 552350fb926a4eedb5ac2690443d8625 (127.2.81.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:45.992108 9148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 552350fb926a4eedb5ac2690443d8625 (127.2.81.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:45.995389 9287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:45.995649 9287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.995954 9518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:45.996212 9518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.997637 9523 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:45.998029 9523 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:46:45.997984 9521 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:45.998322 9521 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:45.998775 9336 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 552350fb926a4eedb5ac2690443d8625 (127.2.81.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:46.002496 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1a8c06366a8492b997e0e594acf767f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:46.002836 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 12:46:46.197595 9517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:46.198022 9517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:46.198161 9517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:46.206486 9517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:46.206631 9517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:46.206761 9517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:46.217518 9517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:45181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=36709
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:46.218806 9517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:46.220041 9517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:46.234036 9528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:46.233997 9529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:46.234015 9531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:46.234881 9517 server_base.cc:1034] running on GCE node
I20260924 12:46:46.235517 9517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:46.236588 9517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:46.237793 9517 hybrid_clock.cc:648] HybridClock initialized: now 1790254006237739 us; error 48 us; skew 500 ppm
I20260924 12:46:46.238113 9517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:46 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:46.241653 9517 init.cc:377] Logged in from keytab as oryx/127.2.81.66@KRBTEST.COM (short username oryx)
I20260924 12:46:46.243219 9517 webserver.cc:460] Webserver started at http://127.2.81.66:36709/ using document root <none> and password file <none>
I20260924 12:46:46.243928 9517 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:46.244063 9517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:46.249831 9517 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 12:46:46.253024 9538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:46.254555 9517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:46:46.254739 9517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "2a26fdf02f374e7c93839150131ea8a5"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "20389b6f1bd8187720536b80c8e8412e"
server_key_iv: "7c5f3a6479617a46a3075197fbd8fa6c"
server_key_version: "encryptionkey@0"
I20260924 12:46:46.255743 9517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:46.279325 9517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:46.285786 9517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:46.286098 9517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:46.287228 9517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:46.288689 9517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:46.288792 9517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:46.288915 9517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:46.289005 9517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:46.324410 9517 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:45181
I20260924 12:46:46.324468 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:45181 every 8 connection(s)
I20260924 12:46:46.326056 9517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:46.332510 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 9517
I20260924 12:46:46.332819 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8867
Sep 24 12:46:46 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
Sep 24 12:46:46 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.124@KRBTEST.COM
Sep 24 12:46:46 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
Sep 24 12:46:46 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.67@KRBTEST.COM
I20260924 12:46:46.357685 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.331766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:54975 (local address 127.2.81.126:33615)
0924 12:46:46.332258 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.332263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.332593 (+ 330us) server_negotiation.cc:407] Connection header received
0924 12:46:46.334990 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.334996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.335088 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:46.335189 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:46.338529 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.339481 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.342815 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.343450 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.343607 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.348799 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:46.348806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:46.348811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:46.348860 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:46.352561 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.354108 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.354115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.354119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.354289 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.355299 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.355304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.355308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.355666 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:46.355828 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:46.357035 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0924 12:46:46.357316 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":337,"threads_started":1}
I20260924 12:46:46.359604 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.329133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:43285 (local address 127.2.81.124:43453)
0924 12:46:46.330289 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.330295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.331271 (+ 976us) server_negotiation.cc:407] Connection header received
0924 12:46:46.335835 (+ 4564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.335840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.335930 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:46.336007 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:46.338964 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.340146 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.342879 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.343525 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.343639 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.349762 (+ 6123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:46.349770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:46.349776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:46.349832 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:46.353383 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.354927 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.354933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.354936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.355099 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.356012 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.356017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.356020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.356367 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:46.356507 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:46.357124 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 12:46:46.357413 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":892,"threads_started":1}
I20260924 12:46:46.359900 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.331457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:54975)
0924 12:46:46.332478 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:46.332494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:46.334505 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:46.335641 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:46.335655 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:46.337240 (+ 1585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:46.338381 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:46.338394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.339609 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.339615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:46.342623 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:46.342642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.343934 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.343940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:46.344304 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.345498 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:46.345529 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:46.348482 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:46.353182 (+ 4700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:46.353194 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:46.353216 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:46.353936 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:46.354978 (+ 1042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:46.354984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:46.354988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:46.355165 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:46.356505 (+ 1340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:46.356510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:46.356673 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 12:46:46.356916 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 12:46:46.357790 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":84,"spinlock_wait_cycles":15104,"thread_start_us":778,"threads_started":1}
I20260924 12:46:46.360965 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.329822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.2.81.66:43285)
0924 12:46:46.330986 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:46.331060 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:46.335313 (+ 4253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:46.336245 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:46.336249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:46.336808 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:46.338733 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:46.338743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.340790 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.340795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:46.342711 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:46.342722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.343941 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.343946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:46.344298 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.345415 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:46.345454 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:46.348482 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:46.353550 (+ 5068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:46.353555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:46.353559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:46.354776 (+ 1217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:46.355715 (+ 939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:46.355719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:46.355722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:46.355877 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:46.356584 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:46.356588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:46.356694 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 12:46:46.359977 (+ 3283us) client_negotiation.cc:240] Negotiation successful
0924 12:46:46.360430 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":406,"spinlock_wait_cycles":71168,"thread_start_us":1040,"threads_started":1}
I20260924 12:46:46.361841 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.329822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.2.81.66:45793)
0924 12:46:46.330986 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:46.331060 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:46.333615 (+ 2555us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x61200002c628. stack: 00007f834692a66e 00007f834692a430 00007f83459b15ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f8346920824 00007f834691fd66 00007f835982d3cd 00007f835982d36c 00007f83597f9b3b 00007f834d32f536 00007f834d32dd9e 00007f834d3b5dca 00007f834d3b3f68 00007f834d3e782f
0924 12:46:46.334507 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:46.335641 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:46.335655 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:46.336720 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:46.337943 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:46.337974 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.339654 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.339662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:46.342815 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:46.342824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.344288 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.344293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:46.344459 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.345415 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:46.345456 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:46.349133 (+ 3677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:46.353827 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:46.353832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:46.353836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:46.354285 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:46.354668 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:46.354674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:46.354678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:46.354986 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:46.355651 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:46.355660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:46.356169 (+ 509us) client_negotiation.cc:769] Sending connection context
0924 12:46:46.360430 (+ 4261us) client_negotiation.cc:240] Negotiation successful
0924 12:46:46.360705 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":136,"spinlock_wait_cycles":2572800,"thread_start_us":486,"threads_started":1}
I20260924 12:46:46.361897 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.329099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:45793 (local address 127.2.81.125:43707)
0924 12:46:46.330438 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.330444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.331211 (+ 767us) server_negotiation.cc:407] Connection header received
0924 12:46:46.334990 (+ 3779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.334996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.335090 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:46.335189 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:46.338176 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.339359 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.342963 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.343590 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.343679 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.349350 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:46.349356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:46.349359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:46.349406 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:46.353676 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.354422 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.354428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.354430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.354553 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.355129 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.355135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.355138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.355511 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:46.355661 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:46.360638 (+ 4977us) server_negotiation.cc:299] Negotiation successful
0924 12:46:46.361044 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":1114,"threads_started":1}
I20260924 12:46:46.362135 9652 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33615
I20260924 12:46:46.362540 9652 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:46.362983 9653 heartbeater.cc:347] Connected to a master server at 127.2.81.125:43707
I20260924 12:46:46.363124 9653 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:46.363457 9652 heartbeater.cc:511] Master 127.2.81.126:33615 requested a full tablet report, sending...
I20260924 12:46:46.363548 9653 heartbeater.cc:511] Master 127.2.81.125:43707 requested a full tablet report, sending...
I20260924 12:46:46.362155 9651 heartbeater.cc:347] Connected to a master server at 127.2.81.124:43453
I20260924 12:46:46.364310 9651 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:46.364699 9651 heartbeater.cc:511] Master 127.2.81.124:43453 requested a full tablet report, sending...
I20260924 12:46:46.365243 9138 ts_manager.cc:194] Registered new tserver with Master: 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181)
I20260924 12:46:46.365594 8539 ts_manager.cc:194] Re-registered known tserver with Master: 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181)
I20260924 12:46:46.365981 9326 ts_manager.cc:194] Registered new tserver with Master: 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181)
I20260924 12:46:46.366693 8539 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:54975
I20260924 12:46:46.368695 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.333846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.2.81.67:42451)
0924 12:46:46.339937 (+ 6091us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.339946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.339974 (+ 28us) server_negotiation.cc:407] Connection header received
0924 12:46:46.340040 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.340048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.340336 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:46.340508 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:46.343935 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.345423 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.348383 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:46.349308 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:46.350633 (+ 1325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:46.356277 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:46.356292 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:46.356306 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:46.356340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:46.359841 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.366103 (+ 6262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.366113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.366116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.366313 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:46.367051 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:46.367056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:46.367059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:46.367423 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:46.367724 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:46.368068 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 12:46:46.368352 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5897,"thread_start_us":4570,"threads_started":1}
I20260924 12:46:46.370321 8990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:46.454389 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8867
I20260924 12:46:46.467510 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:42451
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:33615
--webserver_port=36973
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:46:46.764582 9506 heartbeater.cc:503] Master 127.2.81.126:33615 was elected leader, sending a full tablet report...
W20260924 12:46:46.802158 9664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:46.802446 9664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:46.802577 9664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:46.810025 9664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:46.810145 9664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:46.810297 9664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:46.820559 9664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:42451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=36973
--tserver_master_addrs=127.2.81.126:33615,127.2.81.125:43707,127.2.81.124:43453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:46.821990 9664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:46.823333 9664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:46.837211 9670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:46.837273 9669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:46.837252 9672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:46.838459 9664 server_base.cc:1034] running on GCE node
I20260924 12:46:46.839185 9664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:46.840478 9664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:46.841760 9664 hybrid_clock.cc:648] HybridClock initialized: now 1790254006841693 us; error 58 us; skew 500 ppm
I20260924 12:46:46.842231 9664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:46 dist-test-slave-9tm1 krb5kdc[8494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:46.846258 9664 init.cc:377] Logged in from keytab as oryx/127.2.81.67@KRBTEST.COM (short username oryx)
I20260924 12:46:46.848156 9664 webserver.cc:460] Webserver started at http://127.2.81.67:36973/ using document root <none> and password file <none>
I20260924 12:46:46.848904 9664 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:46.849087 9664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:46.855115 9664 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:46:46.858615 9679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:46.860178 9664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:46:46.860368 9664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "218b188c1ed545c990b93f8fae3a7258"
format_stamp: "Formatted at 2026-09-24 12:46:43 on dist-test-slave-9tm1"
server_key: "b584d54a0d4157d8467beb044d9f6d95"
server_key_iv: "a0430114a8fc5a80cb9ddb16765a663a"
server_key_version: "encryptionkey@0"
I20260924 12:46:46.861630 9664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:46.886291 9664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:46.891955 9664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:46.892208 9664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:46.893276 9664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:46:46.894851 9664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:46.894942 9664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:46.895038 9664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:46.895112 9664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:46.929075 9664 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:42451
I20260924 12:46:46.929203 9791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:42451 every 8 connection(s)
I20260924 12:46:46.930464 9664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:46.938995 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 9664
I20260924 12:46:47.023919 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.960296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.2.81.124:43453)
0924 12:46:46.998357 (+ 38061us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.998365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.998389 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:46:46.998457 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.998462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.998551 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.000601 (+ 2050us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.005885 (+ 5284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.006671 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.008080 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.008688 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.008833 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.016954 (+ 8121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.016961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.016966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.017048 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.020550 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.021407 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.021413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.021417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.021573 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.022025 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.022029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.022033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.022388 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.022613 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.023282 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.023517 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37912,"thread_start_us":26308,"threads_started":1}
Sep 24 12:46:47 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.125@KRBTEST.COM
Sep 24 12:46:47 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.124@KRBTEST.COM
Sep 24 12:46:47 dist-test-slave-9tm1 krb5kdc[8494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254006, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:47.033258 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.959628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.2.81.126:33615)
0924 12:46:46.999665 (+ 40037us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.999671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.999690 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:46:46.999761 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.999765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.999848 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.009368 (+ 9520us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.009568 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.010608 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.014485 (+ 3877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.015135 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.015270 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.020575 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.020582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.020586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.020621 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.027124 (+ 6503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.028872 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.028879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.028882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.029095 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.029574 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.029580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.029583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.029990 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.032284 (+ 2294us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.032490 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.032843 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39916,"thread_start_us":18691,"threads_started":1}
I20260924 12:46:47.034315 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.978869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.2.81.125:43707)
0924 12:46:46.996324 (+ 17455us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.996332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:46.996355 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:46:46.996424 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:46.996430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:46.996525 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:46.996637 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.008114 (+ 11477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.009191 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.011180 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.011807 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.011912 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.021023 (+ 9111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.021031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.021036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.021080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.028394 (+ 7314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.030277 (+ 1883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.030282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.030286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.030462 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.030894 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.030899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.030902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.031389 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.031622 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.032791 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.033098 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17322,"thread_start_us":17882,"threads_started":1}
I20260924 12:46:47.036022 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.000453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.2.81.67:47113)
0924 12:46:47.005515 (+ 5062us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.005534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.005662 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.011871 (+ 6209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.011876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.012675 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:47.013067 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.013077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.015845 (+ 2768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.015851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.018074 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.018093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.021820 (+ 3727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.021827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.022029 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.022540 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:47.022562 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:47.025730 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:47.030185 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:47.030195 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:47.030215 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:47.030941 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:47.031469 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:47.031474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:47.031477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:47.031753 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:47.032523 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:47.032530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:47.032993 (+ 463us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.033315 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.034436 (+ 1121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4925,"mutex_wait_us":4,"thread_start_us":4785,"threads_started":1}
I20260924 12:46:47.036432 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.000578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:47113 (local address 127.2.81.125:43707)
0924 12:46:47.011493 (+ 10915us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.011499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.011517 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:47.011564 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.011570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.011653 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.013219 (+ 1566us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.013442 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.015644 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.018303 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.018969 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.019107 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.026128 (+ 7021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.026137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.026142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.026305 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.029994 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.031105 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.031111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.031114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.031322 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.031938 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.031943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.031946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.032325 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.032450 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.033933 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.034145 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10719,"thread_start_us":10955,"threads_started":1}
I20260924 12:46:47.037807 9794 heartbeater.cc:347] Connected to a master server at 127.2.81.125:43707
I20260924 12:46:47.038166 9794 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:47.038296 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.933492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:38649 (local address 127.2.81.126:33615)
0924 12:46:46.978339 (+ 44847us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.978345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.002523 (+ 24178us) server_negotiation.cc:407] Connection header received
0924 12:46:47.005675 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.005681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.005767 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.005867 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.013269 (+ 7402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.014443 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.019535 (+ 5092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.020195 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.020325 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.029050 (+ 8725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.029059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.029064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.029105 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.033136 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.034326 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.034331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.034335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.034492 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.036000 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.036006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.036010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.036405 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.036702 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.037644 (+ 942us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.037924 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44719,"thread_start_us":65975,"threads_started":1}
I20260924 12:46:47.039126 9794 heartbeater.cc:511] Master 127.2.81.125:43707 requested a full tablet report, sending...
I20260924 12:46:47.039659 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.934110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:38649)
0924 12:46:47.002282 (+ 68172us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.002349 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.005265 (+ 2916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.010338 (+ 5073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.010377 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.011306 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:47.012936 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.012997 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.014983 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.014988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.019310 (+ 4322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.019327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.021832 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.021837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.023534 (+ 1697us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.026079 (+ 2545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:47.026106 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:47.028767 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:47.033357 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:47.033363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:47.033367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:47.033935 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:47.035675 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:47.035680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:47.035684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:47.035869 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:47.036898 (+ 1029us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:47.036903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:47.037120 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.037268 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.037601 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67920,"mutex_wait_us":3512,"thread_start_us":64660,"threads_started":1}
I20260924 12:46:47.041072 9793 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33615
I20260924 12:46:47.041076 9138 ts_manager.cc:194] Registered new tserver with Master: 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451)
I20260924 12:46:47.041214 9793 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:47.041687 9793 heartbeater.cc:511] Master 127.2.81.126:33615 requested a full tablet report, sending...
I20260924 12:46:47.042785 8538 ts_manager.cc:194] Re-registered known tserver with Master: 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451)
I20260924 12:46:47.043013 9798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.934110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.2.81.67:34747)
0924 12:46:47.010816 (+ 76706us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.010843 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.010977 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.012051 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.012056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.013557 (+ 1501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:47.013963 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.013980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.015654 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.015659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.019566 (+ 3907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.019582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.020859 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.020867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.021250 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.022112 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:47.022155 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:47.028303 (+ 6148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:47.038305 (+ 10002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:47.038310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:47.038314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:47.038869 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:47.039343 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:47.039348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:47.039351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:47.039538 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:47.041996 (+ 2458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:47.042001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:47.042150 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.042304 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.042601 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64916,"mutex_wait_us":1,"spinlock_wait_cycles":62336,"thread_start_us":67583,"threads_started":1}
I20260924 12:46:47.043884 8538 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:38649
I20260924 12:46:47.044075 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:46.933433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:34747 (local address 127.2.81.124:43453)
0924 12:46:46.985639 (+ 52206us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:46.985647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.010919 (+ 25272us) server_negotiation.cc:407] Connection header received
0924 12:46:47.011582 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.011587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.011688 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.011808 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.014236 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.015488 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.019717 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.020392 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.020518 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.029092 (+ 8574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.029099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.029104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.029144 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.037399 (+ 8255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.039056 (+ 1657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.039063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.039067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.039232 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.040048 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.040053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.040056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.040465 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.040960 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.042454 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.042699 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52019,"thread_start_us":66989,"threads_started":1}
I20260924 12:46:47.044950 9792 heartbeater.cc:347] Connected to a master server at 127.2.81.124:43453
I20260924 12:46:47.045123 9792 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:47.046002 9792 heartbeater.cc:511] Master 127.2.81.124:43453 requested a full tablet report, sending...
I20260924 12:46:47.047220 9326 ts_manager.cc:194] Registered new tserver with Master: 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451)
I20260924 12:46:47.038605 8539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:47.048722 8539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:47.102914 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.077781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45181 (local address 127.0.0.1:50232)
0924 12:46:47.078458 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.078482 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.078641 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.094629 (+ 15988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.094635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.094674 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:47.095166 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.095180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.097598 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.097603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.099935 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.099955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.102061 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.102067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.102243 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.102304 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.102387 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.102533 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
I20260924 12:46:47.103524 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.079113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.2.81.67:42451)
0924 12:46:47.095268 (+ 16155us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.095274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.095429 (+ 155us) server_negotiation.cc:407] Connection header received
0924 12:46:47.095975 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.096008 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.096406 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.096670 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:47.097488 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.098541 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.100341 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.101762 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.101896 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.102134 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.103014 (+ 880us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.103199 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15980,"thread_start_us":17764,"threads_started":1}
I20260924 12:46:47.103895 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.078480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:42451 (local address 127.0.0.1:40688)
0924 12:46:47.091133 (+ 12653us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.091170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.091358 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.096662 (+ 5304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.096668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.096705 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:47.097191 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.097214 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.098681 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.098687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.100182 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.100196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.102295 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.102302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.102500 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.102533 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.102617 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.102754 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12478,"thread_start_us":23134,"threads_started":1}
I20260924 12:46:47.104493 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.078049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.2.81.66:45181)
0924 12:46:47.093709 (+ 15660us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.093720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.093765 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:46:47.093908 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.093917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.094256 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.094566 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:47.095601 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.097454 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.100223 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.101814 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.103274 (+ 1460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.103512 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.103584 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.104267 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15434,"thread_start_us":16688,"threads_started":1}
I20260924 12:46:47.107613 9586 tablet_service.cc:1467] Processing CreateTablet for tablet cf25abc26cc644159eea518cea1ce7b9 (DEFAULT_TABLE table=test-table [id=ff936ddd691349eb8e1789b56e21c81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:47.107811 9727 tablet_service.cc:1467] Processing CreateTablet for tablet cf25abc26cc644159eea518cea1ce7b9 (DEFAULT_TABLE table=test-table [id=ff936ddd691349eb8e1789b56e21c81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:47.109053 9586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf25abc26cc644159eea518cea1ce7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:47.109232 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf25abc26cc644159eea518cea1ce7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:47.111449 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.078507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:34053 (local address 127.0.0.1:53256)
0924 12:46:47.101900 (+ 23393us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.101921 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.102058 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.103181 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.103186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.103214 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:47.103589 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.103602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.106152 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.106157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.108223 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.108247 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.110615 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.110622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.110815 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.110851 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.110956 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.111106 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23265,"thread_start_us":12970,"threads_started":1}
I20260924 12:46:47.111502 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.079127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.2.81.65:34053)
0924 12:46:47.101462 (+ 22335us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.101472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.102189 (+ 717us) server_negotiation.cc:407] Connection header received
0924 12:46:47.102345 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.102355 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.102655 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.102928 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:47.103777 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.105974 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.108475 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.110383 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.110535 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.110740 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.111022 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.111211 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22141,"thread_start_us":22022,"threads_started":1}
I20260924 12:46:47.116349 9440 tablet_service.cc:1467] Processing CreateTablet for tablet cf25abc26cc644159eea518cea1ce7b9 (DEFAULT_TABLE table=test-table [id=ff936ddd691349eb8e1789b56e21c81b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:47.118644 9440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf25abc26cc644159eea518cea1ce7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:47.123780 9830 tablet_bootstrap.cc:492] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Bootstrap starting.
I20260924 12:46:47.127797 9830 tablet_bootstrap.cc:654] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:47.128791 9831 tablet_bootstrap.cc:492] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Bootstrap starting.
I20260924 12:46:47.129276 9830 log.cc:824] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:47.133165 9831 tablet_bootstrap.cc:654] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:47.133443 9830 tablet_bootstrap.cc:492] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: No bootstrap required, opened a new log
I20260924 12:46:47.133759 9830 ts_tablet_manager.cc:1397] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 12:46:47.134506 9831 log.cc:824] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:47.137710 9834 tablet_bootstrap.cc:492] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Bootstrap starting.
I20260924 12:46:47.138144 9831 tablet_bootstrap.cc:492] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: No bootstrap required, opened a new log
I20260924 12:46:47.138507 9831 ts_tablet_manager.cc:1397] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 12:46:47.141769 9834 tablet_bootstrap.cc:654] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:47.142859 9834 log.cc:824] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:47.145860 9830 raft_consensus.cc:348] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.146497 9830 raft_consensus.cc:374] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:47.146610 9834 tablet_bootstrap.cc:492] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: No bootstrap required, opened a new log
I20260924 12:46:47.146600 9830 raft_consensus.cc:729] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b188c1ed545c990b93f8fae3a7258, State: Initialized, Role: FOLLOWER
I20260924 12:46:47.146925 9834 ts_tablet_manager.cc:1397] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 12:46:47.147297 9830 consensus_queue.cc:260] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.149236 9830 ts_tablet_manager.cc:1428] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260924 12:46:47.149541 9793 heartbeater.cc:503] Master 127.2.81.126:33615 was elected leader, sending a full tablet report...
I20260924 12:46:47.150894 9831 raft_consensus.cc:348] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.151368 9831 raft_consensus.cc:374] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:47.151459 9831 raft_consensus.cc:729] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a26fdf02f374e7c93839150131ea8a5, State: Initialized, Role: FOLLOWER
I20260924 12:46:47.152271 9831 consensus_queue.cc:260] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.158993 9652 heartbeater.cc:503] Master 127.2.81.126:33615 was elected leader, sending a full tablet report...
I20260924 12:46:47.159781 9831 ts_tablet_manager.cc:1428] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20260924 12:46:47.160413 9834 raft_consensus.cc:348] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.160934 9834 raft_consensus.cc:374] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:47.161090 9834 raft_consensus.cc:729] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080d3e378290483b8ea3aa4691656e67, State: Initialized, Role: FOLLOWER
I20260924 12:46:47.161805 9834 consensus_queue.cc:260] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.170989 9834 ts_tablet_manager.cc:1428] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Time spent starting tablet: real 0.024s user 0.016s sys 0.005s
W20260924 12:46:47.183483 9795 tablet.cc:2369] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:47.184901 9836 raft_consensus.cc:482] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:47.185245 9836 raft_consensus.cc:504] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.187064 9836 leader_election.cc:288] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181), 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053)
I20260924 12:46:47.194033 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.187708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:49451 (local address 127.2.81.65:34053)
0924 12:46:47.187894 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.187900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.187918 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:47.187984 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.187988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.188058 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.188143 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:47.188779 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.189650 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.192086 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.192917 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.193097 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.193227 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.193616 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.193755 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:46:47.194067 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.187664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:47175 (local address 127.2.81.66:45181)
0924 12:46:47.187857 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.187864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.188319 (+ 455us) server_negotiation.cc:407] Connection header received
0924 12:46:47.188607 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.188612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.188690 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.188784 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:47.189152 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.189855 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.192080 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.192912 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.193102 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.193199 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.193648 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.193789 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:46:47.194105 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.188059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45181 (local address 127.2.81.67:47175)
0924 12:46:47.188220 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.188237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.188380 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.188807 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.188811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.188848 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:47.189015 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.189027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.189997 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.190001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.191886 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.191900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.193134 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.193141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.193417 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.193459 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.193574 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.193724 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":18,"spinlock_wait_cycles":14976}
I20260924 12:46:47.194555 9798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.187457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:34053 (local address 127.2.81.67:49451)
0924 12:46:47.187657 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:47.187677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:47.187817 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:47.188315 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:47.188320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:47.188373 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:47.188599 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.188611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.189787 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.189792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.191876 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:47.191893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.193123 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.193130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:47.193415 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.193457 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 12:46:47.193542 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 12:46:47.193790 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":161,"spinlock_wait_cycles":22784}
I20260924 12:46:47.195243 9606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf25abc26cc644159eea518cea1ce7b9" candidate_uuid: "218b188c1ed545c990b93f8fae3a7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a26fdf02f374e7c93839150131ea8a5" is_pre_election: true
I20260924 12:46:47.195781 9606 raft_consensus.cc:2454] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218b188c1ed545c990b93f8fae3a7258 in term 0.
I20260924 12:46:47.195770 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf25abc26cc644159eea518cea1ce7b9" candidate_uuid: "218b188c1ed545c990b93f8fae3a7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080d3e378290483b8ea3aa4691656e67" is_pre_election: true
I20260924 12:46:47.196213 9460 raft_consensus.cc:2454] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218b188c1ed545c990b93f8fae3a7258 in term 0.
I20260924 12:46:47.196549 9681 leader_election.cc:302] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218b188c1ed545c990b93f8fae3a7258, 2a26fdf02f374e7c93839150131ea8a5; no voters:
I20260924 12:46:47.197343 9836 raft_consensus.cc:2781] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:47.197513 9836 raft_consensus.cc:482] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:47.197618 9836 raft_consensus.cc:3037] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:47.201102 9836 raft_consensus.cc:504] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.201951 9836 leader_election.cc:288] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [CANDIDATE]: Term 1 election: Requested vote from peers 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181), 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053)
I20260924 12:46:47.202452 9606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf25abc26cc644159eea518cea1ce7b9" candidate_uuid: "218b188c1ed545c990b93f8fae3a7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a26fdf02f374e7c93839150131ea8a5"
I20260924 12:46:47.202454 9460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf25abc26cc644159eea518cea1ce7b9" candidate_uuid: "218b188c1ed545c990b93f8fae3a7258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080d3e378290483b8ea3aa4691656e67"
I20260924 12:46:47.202690 9606 raft_consensus.cc:3037] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:47.202706 9460 raft_consensus.cc:3037] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:47.205049 9606 raft_consensus.cc:2454] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218b188c1ed545c990b93f8fae3a7258 in term 1.
I20260924 12:46:47.205520 9460 raft_consensus.cc:2454] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218b188c1ed545c990b93f8fae3a7258 in term 1.
I20260924 12:46:47.205583 9681 leader_election.cc:302] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218b188c1ed545c990b93f8fae3a7258, 2a26fdf02f374e7c93839150131ea8a5; no voters:
I20260924 12:46:47.205915 9836 raft_consensus.cc:2781] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:47.207015 9836 raft_consensus.cc:686] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 1 LEADER]: Becoming Leader. State: Replica: 218b188c1ed545c990b93f8fae3a7258, State: Running, Role: LEADER
I20260924 12:46:47.207646 9836 consensus_queue.cc:237] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } }
I20260924 12:46:47.214159 8539 catalog_manager.cc:5495] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67). New cstate: current_term: 1 leader_uuid: "218b188c1ed545c990b93f8fae3a7258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } health_report { overall_health: UNKNOWN } } }
W20260924 12:46:47.227919 9508 tablet.cc:2369] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:47.246533 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.240227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.2.81.67:42451)
0924 12:46:47.240418 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.240424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.240463 (+ 39us) server_negotiation.cc:407] Connection header received
0924 12:46:47.240664 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.240668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.240791 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.240929 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:47.241331 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.242248 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.243655 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.244102 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.244199 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.244777 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:47.245524 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:47.245653 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.246093 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.246236 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 12:46:47.255841 9606 raft_consensus.cc:1261] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Refusing update from remote peer 218b188c1ed545c990b93f8fae3a7258: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:47.255841 9460 raft_consensus.cc:1261] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Refusing update from remote peer 218b188c1ed545c990b93f8fae3a7258: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:47.256898 9839 consensus_queue.cc:1035] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:47.257295 9836 consensus_queue.cc:1035] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:47.271102 9842 mvcc.cc:205] Tried to move back new op lower bound from 7332880413710069760 to 7332880413528203264. Current Snapshot: MvccSnapshot[applied={T|T < 7332880413710069760}]
I20260924 12:46:47.289604 8539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:47.289911 8539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:47.290604 9844 mvcc.cc:205] Tried to move back new op lower bound from 7332880413710069760 to 7332880413528203264. Current Snapshot: MvccSnapshot[applied={T|T < 7332880413710069760}]
I20260924 12:46:47.297050 8539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1a8c06366a8492b997e0e594acf767f: Sending DeleteTablet for 3 replicas of tablet cf25abc26cc644159eea518cea1ce7b9
I20260924 12:46:47.298370 9586 tablet_service.cc:1514] Processing DeleteTablet for tablet cf25abc26cc644159eea518cea1ce7b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:47 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:50232
I20260924 12:46:47.298383 9440 tablet_service.cc:1514] Processing DeleteTablet for tablet cf25abc26cc644159eea518cea1ce7b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:47 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:53256
I20260924 12:46:47.298548 9727 tablet_service.cc:1514] Processing DeleteTablet for tablet cf25abc26cc644159eea518cea1ce7b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:47 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:40688
I20260924 12:46:47.311195 9852 tablet_replica.cc:330] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: stopping tablet replica
I20260924 12:46:47.311908 9852 raft_consensus.cc:2229] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:47.312502 9852 raft_consensus.cc:2258] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:47.315069 9852 ts_tablet_manager.cc:1905] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:47.315495 9853 tablet_replica.cc:330] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: stopping tablet replica
I20260924 12:46:47.315966 9853 raft_consensus.cc:2229] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:47.316671 9853 raft_consensus.cc:2258] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:47.319545 9853 ts_tablet_manager.cc:1905] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:47.322889 9852 ts_tablet_manager.cc:1918] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:47.323099 9852 log.cc:1196] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/cf25abc26cc644159eea518cea1ce7b9
I20260924 12:46:47.323727 9852 ts_tablet_manager.cc:1939] T cf25abc26cc644159eea518cea1ce7b9 P 080d3e378290483b8ea3aa4691656e67: Deleting consensus metadata
I20260924 12:46:47.326220 8526 catalog_manager.cc:4841] TS 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053): tablet cf25abc26cc644159eea518cea1ce7b9 (table test-table [id=ff936ddd691349eb8e1789b56e21c81b]) successfully deleted
I20260924 12:46:47.327454 9853 ts_tablet_manager.cc:1918] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:47.327728 9853 log.cc:1196] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/cf25abc26cc644159eea518cea1ce7b9
I20260924 12:46:47.328356 9853 ts_tablet_manager.cc:1939] T cf25abc26cc644159eea518cea1ce7b9 P 218b188c1ed545c990b93f8fae3a7258: Deleting consensus metadata
I20260924 12:46:47.330272 8524 catalog_manager.cc:4841] TS 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451): tablet cf25abc26cc644159eea518cea1ce7b9 (table test-table [id=ff936ddd691349eb8e1789b56e21c81b]) successfully deleted
I20260924 12:46:47.331844 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 8507
I20260924 12:46:47.331840 9851 tablet_replica.cc:330] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: stopping tablet replica
I20260924 12:46:47.333305 8569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:47.334523 9851 raft_consensus.cc:2229] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:47.335115 9851 raft_consensus.cc:2258] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:47.338039 9851 ts_tablet_manager.cc:1905] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:47.351260 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.301925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.2.81.125:43707)
0924 12:46:47.302120 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.302127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.329279 (+ 27152us) server_negotiation.cc:407] Connection header received
0924 12:46:47.332256 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.332263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.332357 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.332423 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.336624 (+ 4201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.337777 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.339872 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.340514 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.340629 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.344320 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.344327 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.344332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.344381 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.347783 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.348742 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.348747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.348751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.348930 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.349485 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.349490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.349493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.349874 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.350084 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.350659 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.350925 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 12:46:47.364794 9851 ts_tablet_manager.cc:1918] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:47.366719 9851 log.cc:1196] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/cf25abc26cc644159eea518cea1ce7b9
I20260924 12:46:47.367765 9851 ts_tablet_manager.cc:1939] T cf25abc26cc644159eea518cea1ce7b9 P 2a26fdf02f374e7c93839150131ea8a5: Deleting consensus metadata
I20260924 12:46:47.372691 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.331063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.2.81.124:43453)
0924 12:46:47.331249 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.331255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.352380 (+ 21125us) server_negotiation.cc:407] Connection header received
0924 12:46:47.352534 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.352540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.352641 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.352771 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.354452 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.355577 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.357513 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.358179 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.358330 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.362389 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.362398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.362403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.362456 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.367384 (+ 4928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.370141 (+ 2757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.370147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.370151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.370339 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.370838 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.370844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.370848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.371225 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.371478 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.372090 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.372357 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:46:47.466225 8524 catalog_manager.cc:4841] TS 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181): tablet cf25abc26cc644159eea518cea1ce7b9 (table test-table [id=ff936ddd691349eb8e1789b56e21c81b]) successfully deleted
I20260924 12:46:47.470299 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 8507
I20260924 12:46:47.514191 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.492544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.2.81.124:43453)
0924 12:46:47.492756 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.492764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.494784 (+ 2020us) server_negotiation.cc:407] Connection header received
0924 12:46:47.495173 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.495181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.495274 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.495351 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.498215 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.499648 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.502158 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.502919 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.503041 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.507301 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.507309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.507313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.507359 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.511079 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.512050 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.512054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.512057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.512209 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.512604 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.512608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.512611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.512901 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.513543 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.513690 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.513933 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:46:47.522303 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:47.493763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.2.81.125:43707)
0924 12:46:47.494003 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:47.494011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:47.505105 (+ 11094us) server_negotiation.cc:407] Connection header received
0924 12:46:47.507269 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:47.507274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:47.507354 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:47.507468 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:47.509024 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.510124 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.511584 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:47.512226 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:47.512355 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:47.516056 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:47.516062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:47.516066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:47.516109 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:47.519092 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.520077 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.520082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.520084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.520243 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:47.520710 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:47.520715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:47.520717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:47.521244 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:47.521492 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:47.521825 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 12:46:47.522023 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 12:46:48.282625 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.281861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:46019)
0924 12:46:48.282277 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.282414 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":214,"threads_started":1}
W20260924 12:46:48.283036 9506 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:33615 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.283972 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.283129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:35425)
0924 12:46:48.283557 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.283712 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":206,"threads_started":1}
W20260924 12:46:48.284479 9793 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:33615 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.286209 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.285505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:48137)
0924 12:46:48.285926 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.286024 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":208,"threads_started":1}
W20260924 12:46:48.286608 9652 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:33615 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.797887 9866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b1a8c06366a8492b997e0e594acf767f)
I20260924 12:46:48.798177 9866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:48.798388 9867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b1a8c06366a8492b997e0e594acf767f)
I20260924 12:46:48.798611 9867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:48.799939 9866 leader_election.cc:288] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615), c6955d68527f48e0b77b0903ad0b08ec (127.2.81.125:43707)
I20260924 12:46:48.799957 9867 leader_election.cc:288] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615), 552350fb926a4eedb5ac2690443d8625 (127.2.81.124:43453)
I20260924 12:46:48.801826 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.800534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37762)
0924 12:46:48.801318 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.801539 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":436,"threads_started":1}
I20260924 12:46:48.803489 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.800145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37750)
0924 12:46:48.803082 (+ 2937us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.803294 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2836,"threads_started":1}
W20260924 12:46:48.804740 9310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
W20260924 12:46:48.805313 9122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.811143 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.800357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:43453 (local address 127.0.0.1:60740)
0924 12:46:48.803768 (+ 3411us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.803785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:48.803929 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:48.804323 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:48.804328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:48.804356 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:48.804595 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:48.804608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.806541 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.806547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:48.808624 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:48.808643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.810357 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.810364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:48.810563 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:48.810605 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 12:46:48.810668 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 12:46:48.810816 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3279,"thread_start_us":3705,"threads_started":1}
I20260924 12:46:48.811401 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.801379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:43707 (local address 127.0.0.1:38352)
0924 12:46:48.802045 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.802064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:48.802218 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:48.805201 (+ 2983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:48.805210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:48.805234 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:48.805461 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:48.805473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.807080 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.807085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:48.809043 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:48.809055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.810670 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.810676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:48.810858 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:48.810891 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:48.810957 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 12:46:48.811099 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":2973,"threads_started":1}
I20260924 12:46:48.812245 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.801824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.2.81.125:43707)
0924 12:46:48.804874 (+ 3050us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:48.804879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:48.804893 (+ 14us) server_negotiation.cc:407] Connection header received
0924 12:46:48.804951 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:48.804955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:48.805047 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:48.805590 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:48.805825 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.806921 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.809241 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.810485 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.811616 (+ 1131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:48.811748 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:48.811797 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 12:46:48.811928 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2932,"thread_start_us":2748,"threads_started":1}
I20260924 12:46:48.812536 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.800386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.2.81.124:43453)
0924 12:46:48.802965 (+ 2579us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:48.802970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:48.803874 (+ 904us) server_negotiation.cc:407] Connection header received
0924 12:46:48.804085 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:48.804090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:48.804169 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:48.804755 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:48.805052 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.806384 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.808857 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:48.810156 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:48.811725 (+ 1569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:48.811850 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:48.811894 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 12:46:48.812017 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2496,"thread_start_us":2366,"threads_started":1}
I20260924 12:46:48.813237 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6955d68527f48e0b77b0903ad0b08ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "552350fb926a4eedb5ac2690443d8625" is_pre_election: true
I20260924 12:46:48.813676 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.813269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37788)
0924 12:46:48.813440 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.813529 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 12:46:48.813695 9148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "552350fb926a4eedb5ac2690443d8625" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c6955d68527f48e0b77b0903ad0b08ec" is_pre_election: true
W20260924 12:46:48.813894 9122 leader_election.cc:334] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.814056 9148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 552350fb926a4eedb5ac2690443d8625 in term 1.
I20260924 12:46:48.814357 9336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6955d68527f48e0b77b0903ad0b08ec in term 1.
I20260924 12:46:48.814563 9310 leader_election.cc:302] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552350fb926a4eedb5ac2690443d8625, c6955d68527f48e0b77b0903ad0b08ec; no voters:
I20260924 12:46:48.814846 9124 leader_election.cc:302] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 552350fb926a4eedb5ac2690443d8625, c6955d68527f48e0b77b0903ad0b08ec; no voters: b1a8c06366a8492b997e0e594acf767f
I20260924 12:46:48.814921 9866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:46:48.815032 9866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b1a8c06366a8492b997e0e594acf767f)
I20260924 12:46:48.815137 9866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:46:48.815141 9867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 12:46:48.815244 9867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Starting leader election (detected failure of leader b1a8c06366a8492b997e0e594acf767f)
I20260924 12:46:48.815342 9867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:46:48.815491 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.815089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37792)
0924 12:46:48.815245 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.815333 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 12:46:48.815755 9310 leader_election.cc:334] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.817407 9867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:48.817589 9866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:48.818106 9867 leader_election.cc:288] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 2 election: Requested vote from peers b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615), 552350fb926a4eedb5ac2690443d8625 (127.2.81.124:43453)
I20260924 12:46:48.818199 9866 leader_election.cc:288] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [CANDIDATE]: Term 2 election: Requested vote from peers b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615), c6955d68527f48e0b77b0903ad0b08ec (127.2.81.125:43707)
I20260924 12:46:48.818691 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.818224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37804)
0924 12:46:48.818448 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.818543 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 12:46:48.818615 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6955d68527f48e0b77b0903ad0b08ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "552350fb926a4eedb5ac2690443d8625"
I20260924 12:46:48.818909 9336 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c6955d68527f48e0b77b0903ad0b08ec in current term 2: Already voted for candidate 552350fb926a4eedb5ac2690443d8625 in this term.
I20260924 12:46:48.819409 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.819027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37810)
0924 12:46:48.819166 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.819262 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 12:46:48.819473 9148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "552350fb926a4eedb5ac2690443d8625" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c6955d68527f48e0b77b0903ad0b08ec"
I20260924 12:46:48.819762 9148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 552350fb926a4eedb5ac2690443d8625 in current term 2: Already voted for candidate c6955d68527f48e0b77b0903ad0b08ec in this term.
I20260924 12:46:48.820034 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.819543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37818)
0924 12:46:48.819790 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.819878 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 12:46:48.820185 9122 leader_election.cc:334] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.820300 9122 leader_election.cc:302] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6955d68527f48e0b77b0903ad0b08ec; no voters: 552350fb926a4eedb5ac2690443d8625, b1a8c06366a8492b997e0e594acf767f
I20260924 12:46:48.820567 9867 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 12:46:48.821512 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:48.821154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:37828)
0924 12:46:48.821314 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:48.821387 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 12:46:48.821663 9310 leader_election.cc:334] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:48.821780 9310 leader_election.cc:302] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 552350fb926a4eedb5ac2690443d8625; no voters: b1a8c06366a8492b997e0e594acf767f, c6955d68527f48e0b77b0903ad0b08ec
I20260924 12:46:48.822022 9866 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 12:46:49.284426 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:49.283717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:50999)
0924 12:46:49.284177 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:49.284264 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260924 12:46:49.286147 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:49.285233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:57471)
0924 12:46:49.285864 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:49.285967 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"threads_started":1}
I20260924 12:46:49.287882 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:49.287207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:53627)
0924 12:46:49.287635 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:49.287719 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":233,"threads_started":1}
I20260924 12:46:50.285959 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.285280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:48889)
0924 12:46:50.285689 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.285774 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":198,"threads_started":1}
I20260924 12:46:50.289040 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.287051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:38771)
0924 12:46:50.288748 (+ 1697us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.288839 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":1486,"threads_started":1}
I20260924 12:46:50.289265 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.288558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:50271)
0924 12:46:50.288961 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.289074 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":203,"threads_started":1}
I20260924 12:46:50.431025 9882 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:50.431242 9882 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:50.431842 9882 leader_election.cc:288] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615), 552350fb926a4eedb5ac2690443d8625 (127.2.81.124:43453)
I20260924 12:46:50.432407 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6955d68527f48e0b77b0903ad0b08ec" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "552350fb926a4eedb5ac2690443d8625" is_pre_election: true
I20260924 12:46:50.432613 9336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6955d68527f48e0b77b0903ad0b08ec in term 2.
I20260924 12:46:50.432833 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.432059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38128)
0924 12:46:50.432521 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.432633 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260924 12:46:50.433082 9124 leader_election.cc:302] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552350fb926a4eedb5ac2690443d8625, c6955d68527f48e0b77b0903ad0b08ec; no voters:
I20260924 12:46:50.433416 9882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 12:46:50.433555 9882 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:50.433660 9882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:46:50.434394 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.433997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38142)
0924 12:46:50.434158 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.434245 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 12:46:50.434600 9122 leader_election.cc:334] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:50.436088 9882 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:50.436684 9882 leader_election.cc:288] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 3 election: Requested vote from peers b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615), 552350fb926a4eedb5ac2690443d8625 (127.2.81.124:43453)
I20260924 12:46:50.437366 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.436895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38144)
0924 12:46:50.437072 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.437167 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 12:46:50.437292 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6955d68527f48e0b77b0903ad0b08ec" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "552350fb926a4eedb5ac2690443d8625"
I20260924 12:46:50.437497 9336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:46:50.438408 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.438038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38150)
0924 12:46:50.438173 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.438263 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 12:46:50.438588 9122 leader_election.cc:334] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
I20260924 12:46:50.440646 9336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6955d68527f48e0b77b0903ad0b08ec in term 3.
I20260924 12:46:50.441174 9124 leader_election.cc:302] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 552350fb926a4eedb5ac2690443d8625, c6955d68527f48e0b77b0903ad0b08ec; no voters: b1a8c06366a8492b997e0e594acf767f
I20260924 12:46:50.441550 9882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 3 FOLLOWER]: Leader election won for term 3
I20260924 12:46:50.441987 9882 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [term 3 LEADER]: Becoming Leader. State: Replica: c6955d68527f48e0b77b0903ad0b08ec, State: Running, Role: LEADER
I20260924 12:46:50.442384 9882 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } }
I20260924 12:46:50.444746 9885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6955d68527f48e0b77b0903ad0b08ec. Latest consensus state: current_term: 3 leader_uuid: "c6955d68527f48e0b77b0903ad0b08ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:50.445087 9885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:50.445663 9887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:50.449142 9887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:50.449956 9887 catalog_manager.cc:1259] Loaded cluster ID: bdf4338ba97f47d087a3c81e17d8f646
I20260924 12:46:50.450027 9887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:50.451115 9887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:50.451918 9887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Loaded TSK: 0
I20260924 12:46:50.452728 9887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:46:50.489601 9137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:50.496812 9137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:50.501339 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.500774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38166)
0924 12:46:50.501021 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.501121 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 12:46:50.501343 9336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [term 3 FOLLOWER]: Refusing update from remote peer c6955d68527f48e0b77b0903ad0b08ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 12:46:50.502002 9885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 12:46:50.502391 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.501980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38168)
0924 12:46:50.502165 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.502259 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 12:46:50.502593 9122 consensus_peers.cc:487] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec -> Peer b1a8c06366a8492b997e0e594acf767f (127.2.81.126:33615): Couldn't send request to peer b1a8c06366a8492b997e0e594acf767f. Status: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 12:46:50.505079 9889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6955d68527f48e0b77b0903ad0b08ec. Latest consensus state: current_term: 3 leader_uuid: "c6955d68527f48e0b77b0903ad0b08ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:50.505389 9889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:46:50.506692 9885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c6955d68527f48e0b77b0903ad0b08ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:50.506976 9885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:50.507956 9890 mvcc.cc:205] Tried to move back new op lower bound from 7332880427006537728 to 7332880426775650304. Current Snapshot: MvccSnapshot[applied={T|T < 7332880427006537728 or (T in {7332880427006537728})}]
I20260924 12:46:50.507454 9889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c6955d68527f48e0b77b0903ad0b08ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b1a8c06366a8492b997e0e594acf767f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33615 } } peers { permanent_uuid: "c6955d68527f48e0b77b0903ad0b08ec" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "552350fb926a4eedb5ac2690443d8625" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 43453 } attrs { promote: false } } }
I20260924 12:46:50.509191 9889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552350fb926a4eedb5ac2690443d8625 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:46:50.529603 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.515727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.2.81.67:42451)
0924 12:46:50.517398 (+ 1671us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.517402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.521446 (+ 4044us) server_negotiation.cc:407] Connection header received
0924 12:46:50.521713 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.521719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.521794 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.521866 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:50.522726 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.523882 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.526532 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.528126 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.528255 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.528396 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.529061 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.529232 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":1438,"threads_started":1}
I20260924 12:46:50.530750 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.515528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:42451 (local address 127.0.0.1:40712)
0924 12:46:50.521328 (+ 5800us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.521355 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:50.521552 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:50.522095 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:50.522099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:50.522123 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:50.522494 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.522509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.524173 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.524180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.526331 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.526346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.528604 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.528612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.528835 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.528871 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:46:50.530165 (+ 1294us) client_negotiation.cc:240] Negotiation successful
0924 12:46:50.530366 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5614,"thread_start_us":5443,"threads_started":1}
I20260924 12:46:50.532123 9727 tablet_service.cc:1467] Processing CreateTablet for tablet abab65ec74d7453bb1fbd75d1bc79bb8 (DEFAULT_TABLE table=test-table [id=2cbb37abd34645f5a726048f9db50926]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:50.532662 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abab65ec74d7453bb1fbd75d1bc79bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:50.535887 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.515034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:34053 (local address 127.0.0.1:53260)
0924 12:46:50.515275 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.515294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:50.515412 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:50.530939 (+ 15527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:50.530944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:50.530980 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:50.531367 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.531380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.532733 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.532738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.534120 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.534134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.535213 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.535219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.535368 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.535391 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 12:46:50.535473 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 12:46:50.535606 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 12:46:50.535959 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.515215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.2.81.65:34053)
0924 12:46:50.530451 (+ 15236us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.530458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.530481 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:46:50.530560 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.530565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.530665 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.531530 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:50.531740 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.532590 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.534286 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.535040 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.535156 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.535261 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.535509 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.535642 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15072,"thread_start_us":15556,"threads_started":1}
I20260924 12:46:50.536747 9440 tablet_service.cc:1467] Processing CreateTablet for tablet abab65ec74d7453bb1fbd75d1bc79bb8 (DEFAULT_TABLE table=test-table [id=2cbb37abd34645f5a726048f9db50926]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:50.537285 9440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abab65ec74d7453bb1fbd75d1bc79bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:50.542425 9900 tablet_bootstrap.cc:492] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Bootstrap starting.
I20260924 12:46:50.544322 9900 tablet_bootstrap.cc:654] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:50.544520 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.521555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45181 (local address 127.0.0.1:50234)
0924 12:46:50.531233 (+ 9678us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.531254 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:50.531372 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:50.538016 (+ 6644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:50.538021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:50.538049 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:50.538194 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.538203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.539618 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.539625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.541812 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.541833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.543668 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.543675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.543904 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.543940 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:46:50.544043 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 12:46:50.544194 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9534,"thread_start_us":9686,"threads_started":1}
I20260924 12:46:50.545593 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.521859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.2.81.66:45181)
0924 12:46:50.537053 (+ 15194us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.537059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.537079 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:46:50.537148 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.537154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.537232 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.537333 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:50.538349 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.539440 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.542031 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.543442 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.543603 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.543718 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.545091 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.545244 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15034,"thread_start_us":15941,"threads_started":1}
I20260924 12:46:50.546303 9586 tablet_service.cc:1467] Processing CreateTablet for tablet abab65ec74d7453bb1fbd75d1bc79bb8 (DEFAULT_TABLE table=test-table [id=2cbb37abd34645f5a726048f9db50926]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:50.546752 9586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abab65ec74d7453bb1fbd75d1bc79bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:50.547436 9901 tablet_bootstrap.cc:492] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Bootstrap starting.
I20260924 12:46:50.547516 9900 tablet_bootstrap.cc:492] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: No bootstrap required, opened a new log
I20260924 12:46:50.547644 9900 ts_tablet_manager.cc:1397] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 12:46:50.548391 9900 raft_consensus.cc:348] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.548666 9900 raft_consensus.cc:374] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:50.548719 9901 tablet_bootstrap.cc:654] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:50.548750 9900 raft_consensus.cc:729] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b188c1ed545c990b93f8fae3a7258, State: Initialized, Role: FOLLOWER
I20260924 12:46:50.549049 9900 consensus_queue.cc:260] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.550159 9900 ts_tablet_manager.cc:1428] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 12:46:50.551257 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.550814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:47991)
0924 12:46:50.551032 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.551117 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 12:46:50.552837 9901 tablet_bootstrap.cc:492] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: No bootstrap required, opened a new log
I20260924 12:46:50.553884 9901 ts_tablet_manager.cc:1397] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 12:46:50.554627 9901 raft_consensus.cc:348] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.554819 9901 raft_consensus.cc:374] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:50.554904 9901 raft_consensus.cc:729] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080d3e378290483b8ea3aa4691656e67, State: Initialized, Role: FOLLOWER
I20260924 12:46:50.555122 9901 consensus_queue.cc:260] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.555395 9794 heartbeater.cc:503] Master 127.2.81.125:43707 was elected leader, sending a full tablet report...
I20260924 12:46:50.555687 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.552301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:60665)
0924 12:46:50.555352 (+ 3051us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.555477 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2899}
I20260924 12:46:50.558314 9905 tablet_bootstrap.cc:492] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Bootstrap starting.
I20260924 12:46:50.558513 9901 ts_tablet_manager.cc:1428] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260924 12:46:50.559262 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.558715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:57837)
0924 12:46:50.558978 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.559107 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260924 12:46:50.560228 9905 tablet_bootstrap.cc:654] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:50.560542 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.560116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:45459)
0924 12:46:50.560316 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.560392 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 12:46:50.560941 9507 heartbeater.cc:503] Master 127.2.81.125:43707 was elected leader, sending a full tablet report...
I20260924 12:46:50.562640 9905 tablet_bootstrap.cc:492] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: No bootstrap required, opened a new log
I20260924 12:46:50.562754 9905 ts_tablet_manager.cc:1397] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 12:46:50.563297 9905 raft_consensus.cc:348] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.563480 9905 raft_consensus.cc:374] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:50.563544 9905 raft_consensus.cc:729] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a26fdf02f374e7c93839150131ea8a5, State: Initialized, Role: FOLLOWER
I20260924 12:46:50.563750 9905 consensus_queue.cc:260] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.564561 9905 ts_tablet_manager.cc:1428] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:50.565305 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.564823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:57399)
0924 12:46:50.565110 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.565190 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260924 12:46:50.566929 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.566017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:59779)
0924 12:46:50.566717 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.566792 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590}
I20260924 12:46:50.566987 9653 heartbeater.cc:503] Master 127.2.81.125:43707 was elected leader, sending a full tablet report...
W20260924 12:46:50.583278 9654 tablet.cc:2369] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:50.827109 9906 raft_consensus.cc:482] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:50.827353 9906 raft_consensus.cc:504] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.828881 9906 leader_election.cc:288] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451), 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181)
I20260924 12:46:50.836103 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.829252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:42451 (local address 127.2.81.65:36783)
0924 12:46:50.829462 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.829479 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:50.829597 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:50.829974 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:50.829980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:50.830009 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:50.830314 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.830323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.831457 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.831461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.833413 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.833431 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.835493 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.835499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.835629 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.835655 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 12:46:50.835743 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 12:46:50.835852 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":43}
I20260924 12:46:50.836720 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.829329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:36783 (local address 127.2.81.67:42451)
0924 12:46:50.829532 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.829538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.829559 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:46:50.829736 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.829741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.829830 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.829934 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:50.830456 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.831312 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.833645 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.835234 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.835389 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.835584 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.836089 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.836475 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:46:50.837785 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.829697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45181 (local address 127.2.81.65:40451)
0924 12:46:50.830499 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.830513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:50.830611 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:50.830952 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:50.830956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:50.830978 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:50.831273 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.831281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.832283 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.832289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.834045 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:50.834056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.837032 (+ 2976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.837045 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:50.837230 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.837264 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:46:50.837373 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 12:46:50.837496 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":27,"thread_start_us":469,"threads_started":1}
I20260924 12:46:50.837915 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.829909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:40451 (local address 127.2.81.66:45181)
0924 12:46:50.830107 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.830113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.830596 (+ 483us) server_negotiation.cc:407] Connection header received
0924 12:46:50.830721 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.830727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.830803 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.830893 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:50.831430 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.832134 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.834196 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.836194 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.836537 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.836637 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.837437 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.837585 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 12:46:50.837816 9747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abab65ec74d7453bb1fbd75d1bc79bb8" candidate_uuid: "080d3e378290483b8ea3aa4691656e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218b188c1ed545c990b93f8fae3a7258" is_pre_election: true
I20260924 12:46:50.838274 9747 raft_consensus.cc:2454] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 080d3e378290483b8ea3aa4691656e67 in term 0.
I20260924 12:46:50.838537 9606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abab65ec74d7453bb1fbd75d1bc79bb8" candidate_uuid: "080d3e378290483b8ea3aa4691656e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a26fdf02f374e7c93839150131ea8a5" is_pre_election: true
I20260924 12:46:50.838886 9606 raft_consensus.cc:2454] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 080d3e378290483b8ea3aa4691656e67 in term 0.
I20260924 12:46:50.838927 9394 leader_election.cc:302] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 080d3e378290483b8ea3aa4691656e67, 218b188c1ed545c990b93f8fae3a7258; no voters:
I20260924 12:46:50.839402 9906 raft_consensus.cc:2781] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:50.839506 9906 raft_consensus.cc:482] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:50.839579 9906 raft_consensus.cc:3037] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:50.841734 9906 raft_consensus.cc:504] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.842310 9906 leader_election.cc:288] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [CANDIDATE]: Term 1 election: Requested vote from peers 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451), 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181)
I20260924 12:46:50.843155 9747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abab65ec74d7453bb1fbd75d1bc79bb8" candidate_uuid: "080d3e378290483b8ea3aa4691656e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218b188c1ed545c990b93f8fae3a7258"
I20260924 12:46:50.843355 9747 raft_consensus.cc:3037] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:50.843286 9606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abab65ec74d7453bb1fbd75d1bc79bb8" candidate_uuid: "080d3e378290483b8ea3aa4691656e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a26fdf02f374e7c93839150131ea8a5"
I20260924 12:46:50.843557 9606 raft_consensus.cc:3037] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:50.845692 9606 raft_consensus.cc:2454] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 080d3e378290483b8ea3aa4691656e67 in term 1.
I20260924 12:46:50.845695 9747 raft_consensus.cc:2454] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 080d3e378290483b8ea3aa4691656e67 in term 1.
I20260924 12:46:50.846220 9394 leader_election.cc:302] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 080d3e378290483b8ea3aa4691656e67, 218b188c1ed545c990b93f8fae3a7258; no voters:
I20260924 12:46:50.846599 9906 raft_consensus.cc:2781] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:50.847201 9906 raft_consensus.cc:686] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 1 LEADER]: Becoming Leader. State: Replica: 080d3e378290483b8ea3aa4691656e67, State: Running, Role: LEADER
I20260924 12:46:50.847623 9906 consensus_queue.cc:237] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } }
I20260924 12:46:50.848235 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.847837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:57053)
0924 12:46:50.847991 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.848097 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 12:46:50.850538 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.849854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:54271)
0924 12:46:50.850258 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.850369 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 12:46:50.851181 9137 catalog_manager.cc:5495] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 080d3e378290483b8ea3aa4691656e67 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "080d3e378290483b8ea3aa4691656e67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "080d3e378290483b8ea3aa4691656e67" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:50.892335 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.887032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.2.81.65:34053)
0924 12:46:50.887219 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.887224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.887242 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:50.887369 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.887374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.887451 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.887586 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:50.887916 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.888616 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.890013 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.890464 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.890561 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.891067 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:50.891662 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:50.891847 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.891987 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.892059 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:46:50.896529 9606 raft_consensus.cc:1261] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Refusing update from remote peer 080d3e378290483b8ea3aa4691656e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:50.896916 9747 raft_consensus.cc:1261] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Refusing update from remote peer 080d3e378290483b8ea3aa4691656e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:50.897217 9910 consensus_queue.cc:1035] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26fdf02f374e7c93839150131ea8a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:50.897795 9906 consensus_queue.cc:1035] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218b188c1ed545c990b93f8fae3a7258" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:50.900022 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.899491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:32879)
0924 12:46:50.899751 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.899852 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 12:46:50.900333 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.899472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:46539)
0924 12:46:50.900072 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.900182 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514}
I20260924 12:46:50.906196 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.905752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:49661)
0924 12:46:50.905909 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.906009 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 12:46:50.906790 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.906124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.66:56273)
0924 12:46:50.906515 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.906620 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260924 12:46:50.906029 9914 mvcc.cc:205] Tried to move back new op lower bound from 7332880428625321984 to 7332880428435689472. Current Snapshot: MvccSnapshot[applied={T|T < 7332880428625321984}]
I20260924 12:46:50.917151 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.916435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.65:49455)
0924 12:46:50.916797 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.916901 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260924 12:46:50.917794 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.916573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.2.81.67:52835)
0924 12:46:50.917476 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.917582 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":775}
I20260924 12:46:50.926030 9137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:50.926326 9137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:50.931006 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.930522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38174)
0924 12:46:50.930724 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.930837 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 12:46:50.932106 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.931658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38180)
0924 12:46:50.931875 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:50.931967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 12:46:50.933025 9137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6955d68527f48e0b77b0903ad0b08ec: Sending DeleteTablet for 3 replicas of tablet abab65ec74d7453bb1fbd75d1bc79bb8
I20260924 12:46:50.933998 9440 tablet_service.cc:1514] Processing DeleteTablet for tablet abab65ec74d7453bb1fbd75d1bc79bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:50 UTC) from {username='oryx', principal='oryx/127.2.81.125@KRBTEST.COM'} at 127.0.0.1:53260
I20260924 12:46:50.934693 9923 tablet_replica.cc:330] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: stopping tablet replica
I20260924 12:46:50.934981 9923 raft_consensus.cc:2229] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:50.935192 9727 tablet_service.cc:1514] Processing DeleteTablet for tablet abab65ec74d7453bb1fbd75d1bc79bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:50 UTC) from {username='oryx', principal='oryx/127.2.81.125@KRBTEST.COM'} at 127.0.0.1:40712
I20260924 12:46:50.935595 9586 tablet_service.cc:1514] Processing DeleteTablet for tablet abab65ec74d7453bb1fbd75d1bc79bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:50 UTC) from {username='oryx', principal='oryx/127.2.81.125@KRBTEST.COM'} at 127.0.0.1:50234
I20260924 12:46:50.935688 9923 raft_consensus.cc:2258] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:50.936792 9923 ts_tablet_manager.cc:1905] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:50.939569 9924 tablet_replica.cc:330] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: stopping tablet replica
I20260924 12:46:50.939883 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 9307
I20260924 12:46:50.940426 9924 raft_consensus.cc:2229] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:50.940706 9924 raft_consensus.cc:2258] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:50.943835 9923 ts_tablet_manager.cc:1918] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:50.944021 9923 log.cc:1196] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/abab65ec74d7453bb1fbd75d1bc79bb8
I20260924 12:46:50.944515 9923 ts_tablet_manager.cc:1939] T abab65ec74d7453bb1fbd75d1bc79bb8 P 080d3e378290483b8ea3aa4691656e67: Deleting consensus metadata
I20260924 12:46:50.945780 9125 catalog_manager.cc:4841] TS 080d3e378290483b8ea3aa4691656e67 (127.2.81.65:34053): tablet abab65ec74d7453bb1fbd75d1bc79bb8 (table test-table [id=2cbb37abd34645f5a726048f9db50926]) successfully deleted
I20260924 12:46:50.949860 9925 tablet_replica.cc:330] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: stopping tablet replica
I20260924 12:46:50.950100 9925 raft_consensus.cc:2229] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:50.950644 9925 raft_consensus.cc:2258] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:50.950776 9924 ts_tablet_manager.cc:1905] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:50.952031 9925 ts_tablet_manager.cc:1905] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:50.957330 9924 ts_tablet_manager.cc:1918] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:50.957513 9924 log.cc:1196] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/abab65ec74d7453bb1fbd75d1bc79bb8
I20260924 12:46:50.958006 9924 ts_tablet_manager.cc:1939] T abab65ec74d7453bb1fbd75d1bc79bb8 P 218b188c1ed545c990b93f8fae3a7258: Deleting consensus metadata
I20260924 12:46:50.958806 9925 ts_tablet_manager.cc:1918] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:50.959198 9925 log.cc:1196] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/abab65ec74d7453bb1fbd75d1bc79bb8
I20260924 12:46:50.959223 9123 catalog_manager.cc:4841] TS 218b188c1ed545c990b93f8fae3a7258 (127.2.81.67:42451): tablet abab65ec74d7453bb1fbd75d1bc79bb8 (table test-table [id=2cbb37abd34645f5a726048f9db50926]) successfully deleted
I20260924 12:46:50.959761 9925 ts_tablet_manager.cc:1939] T abab65ec74d7453bb1fbd75d1bc79bb8 P 2a26fdf02f374e7c93839150131ea8a5: Deleting consensus metadata
I20260924 12:46:50.960808 9123 catalog_manager.cc:4841] TS 2a26fdf02f374e7c93839150131ea8a5 (127.2.81.66:45181): tablet abab65ec74d7453bb1fbd75d1bc79bb8 (table test-table [id=2cbb37abd34645f5a726048f9db50926]) successfully deleted
I20260924 12:46:50.968415 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:50.942272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.2.81.65:34053)
0924 12:46:50.942571 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:50.942577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:50.949785 (+ 7208us) server_negotiation.cc:407] Connection header received
0924 12:46:50.950124 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:50.950186 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:50.950575 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:50.950697 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:50.953132 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.954226 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.956529 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:50.957451 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:50.957594 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:50.960689 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:50.960708 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:50.960729 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:50.960780 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:50.964825 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:50.965792 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:50.965802 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:50.965806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:50.966032 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:50.966591 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:50.966596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:50.966600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:50.967125 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:50.967439 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:50.967767 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 12:46:50.968022 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260924 12:46:50.969226 9500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:51.069492 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 9307
I20260924 12:46:51.084546 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 9517
I20260924 12:46:51.098651 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:51.085339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.2.81.66:45181)
0924 12:46:51.085506 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:51.085510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:51.085528 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:51.085708 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:51.085713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:51.085800 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:51.085922 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:51.087218 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:51.087984 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:51.089327 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:51.089739 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:51.089848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:51.092289 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:51.092304 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:51.092320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:51.092359 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:51.095464 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:51.096351 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:51.096359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:51.096361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:51.096518 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:51.097068 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:51.097075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:51.097077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:51.097474 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:51.097728 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:51.098155 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 12:46:51.098361 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:46:51.099431 9646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:51.191674 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 9517
I20260924 12:46:51.205523 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 9664
I20260924 12:46:51.218107 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:51.206220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.2.81.67:42451)
0924 12:46:51.206390 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:51.206394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:51.206431 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:46:51.206577 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:51.206581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:51.206652 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:51.206737 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:51.207988 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:51.208671 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:51.209903 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:51.210271 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:51.210394 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:51.212816 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:51.212830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:51.212846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:51.212882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:51.215604 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:51.216263 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:51.216270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:51.216273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:51.216407 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:51.216736 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:51.216739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:51.216741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:51.217152 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:51.217362 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:51.217668 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 12:46:51.217824 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 12:46:51.218695 9787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:51.339810 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 9664
I20260924 12:46:51.349912 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:51.349476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38182)
0924 12:46:51.349646 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:51.349740 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 12:46:51.350975 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:51.350627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33615 (local address 127.0.0.1:38184)
0924 12:46:51.350779 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:51.350859 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 12:46:51.357720 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 9000
I20260924 12:46:51.358726 9168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:51.477586 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 9000
W20260924 12:46:51.494613 9312 connection.cc:535] server connection from 127.0.0.1:60740 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:46:51.494964 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 9102
I20260924 12:46:51.495992 9356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:51.609906 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 9102
2026-09-24T12:46:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11309 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:51 dist-test-slave-9tm1 krb5kdc[9934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:51 dist-test-slave-9tm1 krb5kdc[9934](info): set up 2 sockets
Sep 24 12:46:51 dist-test-slave-9tm1 krb5kdc[9934](info): commencing operation
krb5kdc: starting...
W20260924 12:46:52.658717 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:52 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab.
Entry for principal oryx/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab.
I20260924 12:46:52.889812 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:35891
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab
--principal=oryx/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:35891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:53.208719 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:53.209072 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:53.209208 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:53.216409 9947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:53.216535 9947 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 12:46:53.216593 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:53.216675 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:53.216741 9947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:53.216831 9947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:53.226686 9947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:35891
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.126.keytab
--principal=oryx/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:35891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:53.227957 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:53.229301 9947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:53.240522 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:53.240553 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:53.240623 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:53.241330 9947 server_base.cc:1034] running on GCE node
I20260924 12:46:53.242069 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:53.243480 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:53.244691 9947 hybrid_clock.cc:648] HybridClock initialized: now 1790254013244623 us; error 55 us; skew 500 ppm
I20260924 12:46:53.245028 9947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:53 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254013, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:53.248008 9947 init.cc:377] Logged in from keytab as oryx/127.2.81.126@KRBTEST.COM (short username oryx)
I20260924 12:46:53.249464 9947 webserver.cc:460] Webserver started at http://127.2.81.126:33895/ using document root <none> and password file <none>
I20260924 12:46:53.250162 9947 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:53.250247 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:53.250547 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:53.253475 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "477b3ea554d8492c86025c49d8a42311"
format_stamp: "Formatted at 2026-09-24 12:46:53 on dist-test-slave-9tm1"
server_key: "1778a10af2977c70b593e0640f0ed917"
server_key_iv: "2cbedf1b6211d26e252ba5d71aa61097"
server_key_version: "encryptionkey@0"
I20260924 12:46:53.254253 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "477b3ea554d8492c86025c49d8a42311"
format_stamp: "Formatted at 2026-09-24 12:46:53 on dist-test-slave-9tm1"
server_key: "1778a10af2977c70b593e0640f0ed917"
server_key_iv: "2cbedf1b6211d26e252ba5d71aa61097"
server_key_version: "encryptionkey@0"
I20260924 12:46:53.259527 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:53.263556 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:53.265081 9947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:53.265249 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "477b3ea554d8492c86025c49d8a42311"
format_stamp: "Formatted at 2026-09-24 12:46:53 on dist-test-slave-9tm1"
server_key: "1778a10af2977c70b593e0640f0ed917"
server_key_iv: "2cbedf1b6211d26e252ba5d71aa61097"
server_key_version: "encryptionkey@0"
I20260924 12:46:53.265461 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:53.290124 9947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:53.294698 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:53.294929 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:53.315476 9947 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:35891
I20260924 12:46:53.315492 10023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:35891 every 8 connection(s)
I20260924 12:46:53.316804 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:46:53.320447 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 9947
I20260924 12:46:53.320655 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:46:53.321435 10024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:53.321458 2373 external_mini_cluster.cc:1428] Setting key 3d528b20d8bd565a9fb9ca4e2524f33d
I20260924 12:46:53.335695 10024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Bootstrap starting.
Sep 24 12:46:53 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:53.339313 10024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:53.340371 10024 log.cc:824] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:53.343629 10024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: No bootstrap required, opened a new log
I20260924 12:46:53.345541 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:53.323557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.2.81.126:35891)
0924 12:46:53.324369 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:53.324381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:53.324431 (+ 50us) server_negotiation.cc:407] Connection header received
0924 12:46:53.326431 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:53.326456 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:53.326890 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:53.327248 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:53.329016 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.330329 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.332118 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.332910 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.333133 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:53.337876 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:53.337886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:53.337902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:53.337949 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:53.341060 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:53.342170 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:53.342177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:53.342185 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:53.342361 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:53.342775 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:53.342778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:53.342780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:53.343272 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:53.343510 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:53.344006 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 12:46:53.344453 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260924 12:46:53.351424 10024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477b3ea554d8492c86025c49d8a42311" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35891 } }
I20260924 12:46:53.351806 10024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:53.351902 10024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477b3ea554d8492c86025c49d8a42311, State: Initialized, Role: FOLLOWER
I20260924 12:46:53.352424 10024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477b3ea554d8492c86025c49d8a42311" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35891 } }
I20260924 12:46:53.352636 10024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:46:53.352756 10024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:46:53.352895 10024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:53.355165 10024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477b3ea554d8492c86025c49d8a42311" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35891 } }
I20260924 12:46:53.355682 10024 leader_election.cc:302] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 477b3ea554d8492c86025c49d8a42311; no voters:
I20260924 12:46:53.356297 10024 leader_election.cc:288] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:46:53.356454 10029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:53.357497 10029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [term 1 LEADER]: Becoming Leader. State: Replica: 477b3ea554d8492c86025c49d8a42311, State: Running, Role: LEADER
I20260924 12:46:53.358081 10029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477b3ea554d8492c86025c49d8a42311" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35891 } }
I20260924 12:46:53.358518 10024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:46:53.361461 10030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "477b3ea554d8492c86025c49d8a42311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477b3ea554d8492c86025c49d8a42311" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35891 } } }
I20260924 12:46:53.361891 10030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:53.361763 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 477b3ea554d8492c86025c49d8a42311. Latest consensus state: current_term: 1 leader_uuid: "477b3ea554d8492c86025c49d8a42311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477b3ea554d8492c86025c49d8a42311" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 35891 } } }
I20260924 12:46:53.362047 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311 [sys.catalog]: This master's current role is: LEADER
I20260924 12:46:53.362896 10038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:46:53.368678 10038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:46:53.377785 10038 catalog_manager.cc:1347] Generated new cluster ID: 85e1248f2bec4b83b720f1a6ebe98e4b
I20260924 12:46:53.377907 10038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:46:53.396822 10038 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:46:53.398150 10038 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:46:53.403928 10038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Generated new TSK 0
I20260924 12:46:53.404736 10038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab.
Entry for principal oryx/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab.
I20260924 12:46:53.476691 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35891
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:53.812791 10050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:53.813124 10050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:53.813264 10050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:53.820564 10050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:53.820680 10050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:53.820799 10050 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:53.820899 10050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:46:53.830950 10050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:35891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:53.832285 10050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:53.833676 10050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:53.847306 10055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:53.847278 10058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:53.847376 10056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:53.848265 10050 server_base.cc:1034] running on GCE node
I20260924 12:46:53.849191 10050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:53.850405 10050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:53.852027 10050 hybrid_clock.cc:648] HybridClock initialized: now 1790254013851939 us; error 55 us; skew 500 ppm
I20260924 12:46:53.852355 10050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:53 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254013, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:53.855718 10050 init.cc:377] Logged in from keytab as oryx/127.2.81.65@KRBTEST.COM (short username oryx)
I20260924 12:46:53.857313 10050 webserver.cc:460] Webserver started at http://127.2.81.65:34177/ using document root <none> and password file <none>
I20260924 12:46:53.858009 10050 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:53.858099 10050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:53.858422 10050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:53.861584 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "20173d9e12f948688cbddd800c656289"
format_stamp: "Formatted at 2026-09-24 12:46:53 on dist-test-slave-9tm1"
server_key: "5f12579375927aaacf2bafd7020b2fce"
server_key_iv: "6a02d63e80c19bebd73bff7775a409a6"
server_key_version: "encryptionkey@0"
I20260924 12:46:53.862376 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "20173d9e12f948688cbddd800c656289"
format_stamp: "Formatted at 2026-09-24 12:46:53 on dist-test-slave-9tm1"
server_key: "5f12579375927aaacf2bafd7020b2fce"
server_key_iv: "6a02d63e80c19bebd73bff7775a409a6"
server_key_version: "encryptionkey@0"
I20260924 12:46:53.867650 10050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:46:53.871377 10065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:53.872843 10050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:53.873044 10050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "20173d9e12f948688cbddd800c656289"
format_stamp: "Formatted at 2026-09-24 12:46:53 on dist-test-slave-9tm1"
server_key: "5f12579375927aaacf2bafd7020b2fce"
server_key_iv: "6a02d63e80c19bebd73bff7775a409a6"
server_key_version: "encryptionkey@0"
I20260924 12:46:53.873278 10050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:53.912745 10050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:53.917090 10050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:53.917358 10050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:53.920011 10050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:53.920339 10050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:53.920509 10050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:53.920573 10050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:53 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254013, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:53.945274 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:53.920947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:53545 (local address 127.2.81.126:35891)
0924 12:46:53.921689 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:53.921695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:53.923140 (+ 1445us) server_negotiation.cc:407] Connection header received
0924 12:46:53.925353 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:53.925361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:53.925507 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:53.925634 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:53.928155 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.929299 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.932675 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.933237 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.933387 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:53.938247 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:53.938253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:53.938257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:53.938291 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:53.941351 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:53.942469 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:53.942475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:53.942478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:53.942635 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:53.943224 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:53.943228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:53.943230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:53.943520 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:53.943624 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:53.944701 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0924 12:46:53.944959 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":454,"threads_started":1}
I20260924 12:46:53.947165 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:53.922048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.2.81.65:53545)
0924 12:46:53.922924 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:53.922984 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:53.925048 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:53.925921 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:53.925934 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:53.926881 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:53.927894 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:53.927923 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.929573 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.929580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:53.932419 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:53.932444 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.933568 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.933582 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:53.933931 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:53.934816 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:53.934845 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:53.937979 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:53.941531 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:53.941545 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:53.941569 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:53.942345 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:53.942777 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:53.942783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:53.942787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:53.943096 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:53.943641 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:53.943652 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:53.944201 (+ 549us) client_negotiation.cc:769] Sending connection context
0924 12:46:53.944571 (+ 370us) client_negotiation.cc:240] Negotiation successful
0924 12:46:53.945231 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":376,"threads_started":1}
I20260924 12:46:53.968050 10050 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:38599
I20260924 12:46:53.968067 10185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:38599 every 8 connection(s)
I20260924 12:46:53.969492 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:53.953483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:44237 (local address 127.2.81.126:35891)
0924 12:46:53.953674 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:53.953680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:53.954433 (+ 753us) server_negotiation.cc:407] Connection header received
0924 12:46:53.954679 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:53.954685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:53.954754 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:53.954833 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:53.956209 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.957394 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.959781 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.960352 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.960464 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:53.963598 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:53.963604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:53.963609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:53.963652 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:53.966642 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:53.967506 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:53.967510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:53.967513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:53.967641 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:53.967994 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:53.968000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:53.968004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:53.968353 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:53.968502 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:53.968892 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 12:46:53.969134 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:46:53.969527 10050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:46:53.969609 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:53.953558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.2.81.65:44237)
0924 12:46:53.954304 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:53.954324 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:53.954540 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:53.954880 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:53.954885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:53.955502 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:53.955982 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:53.956030 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.957574 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.957579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:53.959598 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:53.959620 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:53.960551 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:53.960558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:53.960749 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:53.961689 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:53.961713 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:53.963369 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:53.966796 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:53.966801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:53.966806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:53.967348 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:53.967750 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:53.967754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:53.967764 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:53.967887 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:53.968460 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:53.968465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:53.968663 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 12:46:53.968782 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 12:46:53.969162 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":410,"threads_started":1}
I20260924 12:46:53.971603 10186 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35891
I20260924 12:46:53.971771 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 10050
I20260924 12:46:53.971894 10186 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:53.971994 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:46:53.972769 10186 heartbeater.cc:511] Master 127.2.81.126:35891 requested a full tablet report, sending...
I20260924 12:46:53.972805 2373 external_mini_cluster.cc:1428] Setting key 75387db95fb85080e50185fd282105e4
I20260924 12:46:53.975154 9979 ts_manager.cc:194] Registered new tserver with Master: 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599)
I20260924 12:46:53.977222 9979 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:53545
WARNING: no policy specified for oryx/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab.
Entry for principal oryx/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab.
I20260924 12:46:54.020318 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35891
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:54.345019 10190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:54.345342 10190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:54.345475 10190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:54.352995 10190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:54.353139 10190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:54.353253 10190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:54.353349 10190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:46:54.364449 10190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:35891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:54.365849 10190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:54.367230 10190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:54.381347 10195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:54.381325 10198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:54.381297 10196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:54.382084 10190 server_base.cc:1034] running on GCE node
I20260924 12:46:54.382761 10190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:54.383757 10190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:54.385056 10190 hybrid_clock.cc:648] HybridClock initialized: now 1790254014384944 us; error 51 us; skew 500 ppm
I20260924 12:46:54.385376 10190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:54 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254014, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:54.388654 10190 init.cc:377] Logged in from keytab as oryx/127.2.81.66@KRBTEST.COM (short username oryx)
I20260924 12:46:54.390316 10190 webserver.cc:460] Webserver started at http://127.2.81.66:46351/ using document root <none> and password file <none>
I20260924 12:46:54.391009 10190 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:54.391113 10190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:54.391439 10190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:54.394519 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "4b414ee801ab4e198b298a99db0f6af0"
format_stamp: "Formatted at 2026-09-24 12:46:54 on dist-test-slave-9tm1"
server_key: "d8ca87f34b07b6d2498ab6784a3fff2b"
server_key_iv: "5e7a35a904b098ff1f53455b72373238"
server_key_version: "encryptionkey@0"
I20260924 12:46:54.395345 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "4b414ee801ab4e198b298a99db0f6af0"
format_stamp: "Formatted at 2026-09-24 12:46:54 on dist-test-slave-9tm1"
server_key: "d8ca87f34b07b6d2498ab6784a3fff2b"
server_key_iv: "5e7a35a904b098ff1f53455b72373238"
server_key_version: "encryptionkey@0"
I20260924 12:46:54.400918 10190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:46:54.404773 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:54.406273 10190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:54.406478 10190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "4b414ee801ab4e198b298a99db0f6af0"
format_stamp: "Formatted at 2026-09-24 12:46:54 on dist-test-slave-9tm1"
server_key: "d8ca87f34b07b6d2498ab6784a3fff2b"
server_key_iv: "5e7a35a904b098ff1f53455b72373238"
server_key_version: "encryptionkey@0"
I20260924 12:46:54.406695 10190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:54.425048 10190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:54.429462 10190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:54.429728 10190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:54.432463 10190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:54.432554 10190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:54.432677 10190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:54.432732 10190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:54 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254014, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:54.455824 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:54.433169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:45987 (local address 127.2.81.126:35891)
0924 12:46:54.433377 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:54.433383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:54.434854 (+ 1471us) server_negotiation.cc:407] Connection header received
0924 12:46:54.437515 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:54.437522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:54.437616 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:54.437719 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:54.440030 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.441219 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.443567 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.444159 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.444315 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:54.449660 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:54.449666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:54.449669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:54.449707 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:54.452432 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:54.453388 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:54.453394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:54.453398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:54.453533 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:54.454014 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:54.454018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:54.454020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:54.454320 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:54.454476 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:54.455238 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 12:46:54.455486 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 12:46:54.457252 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:54.433744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.2.81.66:45987)
0924 12:46:54.434644 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:54.434698 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:54.436821 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:54.438020 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:54.438036 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:54.438868 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:54.439770 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:54.439816 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.441419 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.441428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:54.443408 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:54.443423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.444483 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.444491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:54.444837 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:54.445787 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:54.445827 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:54.449398 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:54.452651 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:54.452660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:54.452676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:54.453244 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:54.453647 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:54.453652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:54.453655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:54.453884 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:54.454458 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:54.454466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:54.454859 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 12:46:54.455103 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 12:46:54.455664 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":358,"threads_started":1}
I20260924 12:46:54.477932 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:54.463274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.2.81.66:52251)
0924 12:46:54.464005 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:54.464028 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:54.464182 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:54.464521 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:54.464526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:54.465195 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:54.465676 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:54.465691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.466701 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.466707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:54.467713 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:54.467726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.468480 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.468486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:54.468681 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:54.469558 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:54.469591 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:54.471046 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:54.474514 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:54.474521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:54.474526 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:54.475125 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:54.475579 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:54.475587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:54.475592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:54.475733 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:54.476578 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:54.476585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:54.476832 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 12:46:54.477038 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 12:46:54.477423 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260924 12:46:54.478664 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:54.463207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:52251 (local address 127.2.81.126:35891)
0924 12:46:54.463376 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:54.463381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:54.464115 (+ 734us) server_negotiation.cc:407] Connection header received
0924 12:46:54.464310 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:54.464316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:54.464394 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:54.464478 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:54.465851 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.466564 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.467865 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:54.468272 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:54.468383 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:54.471362 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:54.471370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:54.471376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:54.471417 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:54.474334 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:54.475281 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:54.475286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:54.475288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:54.475434 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:54.475866 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:54.475872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:54.475876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:54.476332 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:54.476521 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:54.478272 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0924 12:46:54.478453 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 12:46:54.487718 10190 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:35653
I20260924 12:46:54.487737 10324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:35653 every 8 connection(s)
I20260924 12:46:54.489051 10190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:46:54.491050 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 10190
I20260924 12:46:54.491066 10325 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35891
I20260924 12:46:54.491205 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:46:54.491290 10325 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:54.491652 2373 external_mini_cluster.cc:1428] Setting key f2e0add9612d9cf863a09c526015d501
I20260924 12:46:54.492007 10325 heartbeater.cc:511] Master 127.2.81.126:35891 requested a full tablet report, sending...
I20260924 12:46:54.493445 9978 ts_manager.cc:194] Registered new tserver with Master: 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:54.494241 9978 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:45987
WARNING: no policy specified for oryx/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab.
Entry for principal oryx/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab.
I20260924 12:46:54.537328 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:35891
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:54.896816 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:54.897176 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:54.897316 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:54.905205 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:54.905332 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:54.905447 10329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 12:46:54.905565 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:46:54.916129 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.81.126:35891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:54.917501 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:54.918869 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:54.932319 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:54.932493 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:54.932518 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:54.933007 10329 server_base.cc:1034] running on GCE node
I20260924 12:46:54.933653 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:54.934640 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:54.935900 10329 hybrid_clock.cc:648] HybridClock initialized: now 1790254014935800 us; error 63 us; skew 500 ppm
I20260924 12:46:54.936240 10329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:46:54 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254014, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:46:54.939725 10329 init.cc:377] Logged in from keytab as oryx/127.2.81.67@KRBTEST.COM (short username oryx)
I20260924 12:46:54.941386 10329 webserver.cc:460] Webserver started at http://127.2.81.67:37157/ using document root <none> and password file <none>
I20260924 12:46:54.942102 10329 fs_manager.cc:359] Metadata directory not provided
I20260924 12:46:54.942210 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:46:54.942543 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:46:54.945657 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "7f779de13b94419db3fdedba7d249d5e"
format_stamp: "Formatted at 2026-09-24 12:46:54 on dist-test-slave-9tm1"
server_key: "dd7f472cacd569e109dc25d77a7a9dea"
server_key_iv: "c832cf71bf7aeefd2ff0aef60c0a4bf0"
server_key_version: "encryptionkey@0"
I20260924 12:46:54.946688 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "7f779de13b94419db3fdedba7d249d5e"
format_stamp: "Formatted at 2026-09-24 12:46:54 on dist-test-slave-9tm1"
server_key: "dd7f472cacd569e109dc25d77a7a9dea"
server_key_iv: "c832cf71bf7aeefd2ff0aef60c0a4bf0"
server_key_version: "encryptionkey@0"
I20260924 12:46:54.955102 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260924 12:46:54.959518 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:54.961410 10329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 12:46:54.961596 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "7f779de13b94419db3fdedba7d249d5e"
format_stamp: "Formatted at 2026-09-24 12:46:54 on dist-test-slave-9tm1"
server_key: "dd7f472cacd569e109dc25d77a7a9dea"
server_key_iv: "c832cf71bf7aeefd2ff0aef60c0a4bf0"
server_key_version: "encryptionkey@0"
I20260924 12:46:54.961825 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:46:54.980528 10186 heartbeater.cc:503] Master 127.2.81.126:35891 was elected leader, sending a full tablet report...
I20260924 12:46:54.986140 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:46:54.991276 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:46:54.991575 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:46:54.994304 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:46:54.994410 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:46:54.994534 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:46:54.994589 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:46:55 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254014, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:55.021055 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:54.995550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.2.81.67:46687)
0924 12:46:54.996448 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:54.996498 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:54.998578 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:54.999825 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:54.999838 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.000671 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:55.001690 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.001716 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.003114 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.003124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.005298 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.005319 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.006357 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.006367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.006763 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.007730 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:55.007781 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:55.011566 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:55.015572 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:55.015585 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:55.015609 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:55.016401 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:55.016893 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:55.016899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:55.016903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:55.017265 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:55.017944 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:55.017958 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:55.018527 (+ 569us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.018901 (+ 374us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.019518 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":365,"threads_started":1}
I20260924 12:46:55.022356 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:54.994857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:46687 (local address 127.2.81.126:35891)
0924 12:46:54.995436 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:54.995442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:54.996641 (+ 1199us) server_negotiation.cc:407] Connection header received
0924 12:46:54.999063 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:54.999070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:54.999158 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:54.999242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:55.001915 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.002894 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.005483 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.006046 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.006195 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.011963 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:55.011971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:55.011976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:55.012023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:55.015365 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.016581 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.016588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.016592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.016753 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.017407 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.017411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.017413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.017787 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:55.017959 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.021690 (+ 3731us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.021963 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":338,"threads_started":1}
I20260924 12:46:55.044432 10329 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:41527
I20260924 12:46:55.044448 10464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:41527 every 8 connection(s)
I20260924 12:46:55.045837 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:46:55.048015 10465 heartbeater.cc:347] Connected to a master server at 127.2.81.126:35891
I20260924 12:46:55.048050 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.034369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.2.81.67:40069)
0924 12:46:55.035104 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.035128 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.035318 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.035669 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.035674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.036343 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:55.036775 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.036800 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.038164 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.038168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.039345 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.039359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.040159 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.040165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.040325 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.040952 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:55.040967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:55.042090 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:55.045279 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:55.045284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:55.045289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:55.045901 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:55.046304 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:55.046308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:55.046314 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:55.046443 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:55.047087 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:55.047092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:55.047285 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.047406 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.047686 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":335,"threads_started":1}
I20260924 12:46:55.048246 10465 heartbeater.cc:464] Registering TS with master...
I20260924 12:46:55.048776 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.034419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:40069 (local address 127.2.81.126:35891)
0924 12:46:55.034620 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.034627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.035241 (+ 614us) server_negotiation.cc:407] Connection header received
0924 12:46:55.035443 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.035447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.035514 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.035609 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:55.037037 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.038010 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.039483 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.039974 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.040126 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.042448 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:55.042455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:55.042460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:55.042505 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:55.045150 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.046057 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.046061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.046063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.046189 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.046552 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.046558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.046561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.046936 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:55.048200 (+ 1264us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.048322 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.048554 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:55.048950 10465 heartbeater.cc:511] Master 127.2.81.126:35891 requested a full tablet report, sending...
I20260924 12:46:55.050100 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 10329
I20260924 12:46:55.050247 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:46:55.050302 9978 ts_manager.cc:194] Registered new tserver with Master: 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527)
I20260924 12:46:55.050752 2373 external_mini_cluster.cc:1428] Setting key f7556d0686ff43cb23f60ffd5050b7c0
I20260924 12:46:55.051182 9978 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:46687
I20260924 12:46:55.053377 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 12:46:55 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 12:46:55 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:46:55 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 12:46:55.102190 10353 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:55.090970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.2.81.126:35891)
0924 12:46:55.091225 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.091230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.091386 (+ 156us) server_negotiation.cc:407] Connection header received
0924 12:46:55.091473 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.091477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.091573 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.091672 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:55.093264 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.094320 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.095646 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.096094 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.096196 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.101888 (+ 5692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52916: BlockingRecv error: recv got EOF from 127.0.0.1:52916 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 24 12:46:55 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254015, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:55.118366 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.105369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.2.81.126:35891)
0924 12:46:55.105589 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.105595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.105815 (+ 220us) server_negotiation.cc:407] Connection header received
0924 12:46:55.105998 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.106003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.106065 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.106148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:55.107300 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.108015 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.109258 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.109705 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.109819 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.113515 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:55.113520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:55.113524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:55.113563 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:55.115850 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.116535 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.116541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.116545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.116663 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.117030 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.117034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.117036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.117313 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:55.117557 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.117849 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.118073 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 12:46:55.122680 9978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:55.129887 9978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:55.161024 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.142471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41527 (local address 127.0.0.1:33628)
0924 12:46:55.149816 (+ 7345us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.149874 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.150137 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.152951 (+ 2814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.152960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.153020 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.153512 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.153524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.155721 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.155725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.158140 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.158149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.159900 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.159906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.160170 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.160228 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.160348 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.160541 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7037,"mutex_wait_us":143,"spinlock_wait_cycles":768,"thread_start_us":6484,"threads_started":1}
I20260924 12:46:55.161226 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.142488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:38599 (local address 127.0.0.1:53144)
0924 12:46:55.151349 (+ 8861us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.151371 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.151517 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.153043 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.153046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.153099 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.153746 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.153756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.155778 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.155783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.157924 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.157941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.160260 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.160266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.160485 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.160544 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.160630 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.160783 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8726,"mutex_wait_us":59,"spinlock_wait_cycles":4224,"thread_start_us":10102,"threads_started":1}
I20260924 12:46:55.161712 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.143856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.2.81.66:35653)
0924 12:46:55.150886 (+ 7030us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.150897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.153016 (+ 2119us) server_negotiation.cc:407] Connection header received
0924 12:46:55.153325 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.153331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.153530 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.153695 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.154349 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.155260 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.158312 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.159720 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.159829 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.160047 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.160788 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.160956 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6850,"thread_start_us":6727,"threads_started":1}
I20260924 12:46:55.161890 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.142980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.2.81.67:41527)
0924 12:46:55.152183 (+ 9203us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.152194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.152240 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:46:55.152350 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.152358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.152644 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.153531 (+ 887us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.153871 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.155586 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.158308 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.159677 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.161075 (+ 1398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.161309 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.161392 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.161593 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8976,"thread_start_us":8801,"threads_started":1}
I20260924 12:46:55.161922 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.143295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35653 (local address 127.0.0.1:46270)
0924 12:46:55.152875 (+ 9580us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.152892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.153125 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.153920 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.153923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.153942 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.154200 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.154206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.155399 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.155403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.158096 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.158111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.160223 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.160231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.160488 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.160549 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.160630 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.161529 (+ 899us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9420,"mutex_wait_us":868,"spinlock_wait_cycles":6912,"thread_start_us":7752,"threads_started":1}
I20260924 12:46:55.162483 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.142856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.2.81.65:38599)
0924 12:46:55.152512 (+ 9656us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.152519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.152543 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:46:55.152625 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.152632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.152830 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.153928 (+ 1098us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.154116 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.155589 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.158099 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.159698 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.159837 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.160047 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.162162 (+ 2115us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.162292 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9508,"thread_start_us":9360,"threads_started":1}
I20260924 12:46:55.165586 10255 tablet_service.cc:1467] Processing CreateTablet for tablet f750aa46a44f43a7934641028dbb358f (DEFAULT_TABLE table=test-table [id=6546ace19d23472397a5f2096fb6e0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:55.165596 10395 tablet_service.cc:1467] Processing CreateTablet for tablet f750aa46a44f43a7934641028dbb358f (DEFAULT_TABLE table=test-table [id=6546ace19d23472397a5f2096fb6e0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:55.165587 10116 tablet_service.cc:1467] Processing CreateTablet for tablet f750aa46a44f43a7934641028dbb358f (DEFAULT_TABLE table=test-table [id=6546ace19d23472397a5f2096fb6e0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:55.166823 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f750aa46a44f43a7934641028dbb358f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.166851 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f750aa46a44f43a7934641028dbb358f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.166851 10395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f750aa46a44f43a7934641028dbb358f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.178718 10489 tablet_bootstrap.cc:492] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Bootstrap starting.
I20260924 12:46:55.181798 10491 tablet_bootstrap.cc:492] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Bootstrap starting.
I20260924 12:46:55.182869 10489 tablet_bootstrap.cc:654] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.184388 10489 log.cc:824] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:55.185592 10491 tablet_bootstrap.cc:654] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.186017 10490 tablet_bootstrap.cc:492] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Bootstrap starting.
I20260924 12:46:55.187544 10491 log.cc:824] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:55.190495 10490 tablet_bootstrap.cc:654] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.190640 10489 tablet_bootstrap.cc:492] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: No bootstrap required, opened a new log
I20260924 12:46:55.190907 10489 ts_tablet_manager.cc:1397] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260924 12:46:55.190979 10491 tablet_bootstrap.cc:492] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: No bootstrap required, opened a new log
I20260924 12:46:55.191275 10491 ts_tablet_manager.cc:1397] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 12:46:55.191826 10490 log.cc:824] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:46:55.200131 10490 tablet_bootstrap.cc:492] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: No bootstrap required, opened a new log
I20260924 12:46:55.200057 10489 raft_consensus.cc:348] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.200448 10490 ts_tablet_manager.cc:1397] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260924 12:46:55.200479 10489 raft_consensus.cc:374] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.200570 10489 raft_consensus.cc:729] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b414ee801ab4e198b298a99db0f6af0, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.201187 10489 consensus_queue.cc:260] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.201730 10491 raft_consensus.cc:348] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.202165 10491 raft_consensus.cc:374] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.202196 10325 heartbeater.cc:503] Master 127.2.81.126:35891 was elected leader, sending a full tablet report...
I20260924 12:46:55.202256 10491 raft_consensus.cc:729] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20173d9e12f948688cbddd800c656289, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.202961 10491 consensus_queue.cc:260] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.206048 10489 ts_tablet_manager.cc:1428] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Time spent starting tablet: real 0.015s user 0.009s sys 0.002s
I20260924 12:46:55.210690 10491 ts_tablet_manager.cc:1428] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Time spent starting tablet: real 0.019s user 0.017s sys 0.004s
I20260924 12:46:55.211694 10490 raft_consensus.cc:348] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.212095 10490 raft_consensus.cc:374] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.212206 10490 raft_consensus.cc:729] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.212815 10490 consensus_queue.cc:260] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.214102 10465 heartbeater.cc:503] Master 127.2.81.126:35891 was elected leader, sending a full tablet report...
I20260924 12:46:55.214774 10490 ts_tablet_manager.cc:1428] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
W20260924 12:46:55.222019 10187 tablet.cc:2369] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:55.241155 10326 tablet.cc:2369] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:46:55.297597 10466 tablet.cc:2369] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:46:55.428791 10496 raft_consensus.cc:482] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:55.429191 10496 raft_consensus.cc:504] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.430879 10496 leader_election.cc:288] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:55.435461 10495 raft_consensus.cc:482] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:55.435810 10495 raft_consensus.cc:504] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.437650 10495 leader_election.cc:288] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527), 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599)
I20260924 12:46:55.440543 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.431381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:53937 (local address 127.2.81.67:41527)
0924 12:46:55.431569 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.431574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.432564 (+ 990us) server_negotiation.cc:407] Connection header received
0924 12:46:55.432645 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.432650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.432723 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.432804 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.433650 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.434737 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.437954 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.439280 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.439405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.439519 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.440104 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.440240 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:46:55.441607 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.431379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41527 (local address 127.2.81.65:53937)
0924 12:46:55.432420 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.432435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.432545 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.432844 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.432858 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.432879 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.433429 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.433440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.434932 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.434939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.437707 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.437733 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.439722 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.439729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.439905 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.439938 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.440932 (+ 994us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.441135 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":449,"thread_start_us":529,"threads_started":1}
I20260924 12:46:55.442839 10415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f750aa46a44f43a7934641028dbb358f" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f779de13b94419db3fdedba7d249d5e" is_pre_election: true
I20260924 12:46:55.443542 10415 raft_consensus.cc:2454] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 0.
I20260924 12:46:55.444555 10067 leader_election.cc:302] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:55.445152 10496 raft_consensus.cc:2781] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:55.445359 10496 raft_consensus.cc:482] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:55.445465 10496 raft_consensus.cc:3037] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.447947 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.431380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:57755 (local address 127.2.81.66:35653)
0924 12:46:55.431580 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.431586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.432231 (+ 645us) server_negotiation.cc:407] Connection header received
0924 12:46:55.432437 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.432441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.432577 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.432690 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.433605 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.434799 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.437961 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.440187 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.440297 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.440400 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.447354 (+ 6954us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.447531 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:46:55.448832 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.431384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35653 (local address 127.2.81.65:57755)
0924 12:46:55.432112 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.432129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.432298 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.432753 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.432756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.432783 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.433331 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.433349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.434932 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.434939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.437725 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.437733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.446208 (+ 8475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.446218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.446446 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.446483 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.448234 (+ 1751us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.448402 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":92,"thread_start_us":385,"threads_started":1}
I20260924 12:46:55.449353 10496 raft_consensus.cc:504] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.450773 10415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f750aa46a44f43a7934641028dbb358f" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f779de13b94419db3fdedba7d249d5e"
I20260924 12:46:55.451043 10415 raft_consensus.cc:3037] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.455099 10415 raft_consensus.cc:2454] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 1.
I20260924 12:46:55.455614 10067 leader_election.cc:302] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:55.456048 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f750aa46a44f43a7934641028dbb358f" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0" is_pre_election: true
I20260924 12:46:55.456646 10280 raft_consensus.cc:2454] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 0.
I20260924 12:46:55.456939 10496 leader_election.cc:288] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 election: Requested vote from peers 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:55.457162 10496 raft_consensus.cc:2781] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:55.457342 10496 raft_consensus.cc:686] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 1 LEADER]: Becoming Leader. State: Replica: 20173d9e12f948688cbddd800c656289, State: Running, Role: LEADER
I20260924 12:46:55.457931 10496 consensus_queue.cc:237] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } }
I20260924 12:46:55.459074 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f750aa46a44f43a7934641028dbb358f" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0"
I20260924 12:46:55.459282 10280 raft_consensus.cc:3037] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.462677 10280 raft_consensus.cc:2454] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 1.
I20260924 12:46:55.462977 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.438555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44543 (local address 127.2.81.67:41527)
0924 12:46:55.440885 (+ 2330us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.440891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.452744 (+ 11853us) server_negotiation.cc:407] Connection header received
0924 12:46:55.454945 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.454951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.455028 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.455109 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.456675 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.457725 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.460530 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.461723 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.461836 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.461942 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.462531 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.462683 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2223,"thread_start_us":14331,"threads_started":1}
I20260924 12:46:55.463322 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.438611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:43595 (local address 127.2.81.65:38599)
0924 12:46:55.438848 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.438856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.454444 (+ 15588us) server_negotiation.cc:407] Connection header received
0924 12:46:55.454766 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.454773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.454861 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.454925 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.456295 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.457423 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.460277 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.461597 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.461743 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.461870 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.462878 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.463052 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:46:55.464064 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.438258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:38599 (local address 127.2.81.66:43595)
0924 12:46:55.454266 (+ 16008us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.454285 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.454573 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.455133 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.455137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.455164 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.456038 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.456054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.457600 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.457605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.460072 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.460088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.462457 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.462465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.462670 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.462702 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.463467 (+ 765us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.463623 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15727,"mutex_wait_us":38,"thread_start_us":15964,"threads_started":1}
I20260924 12:46:55.464088 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.437914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41527 (local address 127.2.81.66:44543)
0924 12:46:55.452588 (+ 14674us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.452616 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.454762 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.455944 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.455948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.455981 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.456491 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.456506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.457922 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.457927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.460338 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.460357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.462165 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.462172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.462335 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.462368 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.463304 (+ 936us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.463603 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14497,"mutex_wait_us":345,"thread_start_us":14849,"threads_started":1}
I20260924 12:46:55.464762 10415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f750aa46a44f43a7934641028dbb358f" candidate_uuid: "4b414ee801ab4e198b298a99db0f6af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f779de13b94419db3fdedba7d249d5e" is_pre_election: true
I20260924 12:46:55.465124 10415 raft_consensus.cc:2379] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4b414ee801ab4e198b298a99db0f6af0 in current term 1: Already voted for candidate 20173d9e12f948688cbddd800c656289 in this term.
I20260924 12:46:55.465164 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f750aa46a44f43a7934641028dbb358f" candidate_uuid: "4b414ee801ab4e198b298a99db0f6af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289" is_pre_election: true
I20260924 12:46:55.466274 10206 leader_election.cc:302] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b414ee801ab4e198b298a99db0f6af0; no voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e
I20260924 12:46:55.466691 10495 raft_consensus.cc:2726] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 12:46:55.467509 9979 catalog_manager.cc:5495] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20173d9e12f948688cbddd800c656289 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "20173d9e12f948688cbddd800c656289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:55.532629 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.527588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.2.81.65:38599)
0924 12:46:55.527755 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.527760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.527780 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:46:55.527895 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.527901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.527980 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.528111 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:55.528461 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.529217 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.530502 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.530914 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.531023 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.531466 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:55.532041 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:55.532168 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.532264 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.532347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 12:46:55.541163 10415 raft_consensus.cc:1261] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Refusing update from remote peer 20173d9e12f948688cbddd800c656289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:55.541203 10280 raft_consensus.cc:1261] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Refusing update from remote peer 20173d9e12f948688cbddd800c656289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:55.542152 10503 consensus_queue.cc:1035] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:55.542502 10496 consensus_queue.cc:1035] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:55.555223 10506 mvcc.cc:205] Tried to move back new op lower bound from 7332880447645769728 to 7332880447322824704. Current Snapshot: MvccSnapshot[applied={T|T < 7332880447645769728}]
I20260924 12:46:55.557021 10507 mvcc.cc:205] Tried to move back new op lower bound from 7332880447645769728 to 7332880447322824704. Current Snapshot: MvccSnapshot[applied={T|T < 7332880447645769728}]
I20260924 12:46:55.568756 9978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:55.569046 9978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:55.572572 9978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Sending DeleteTablet for 3 replicas of tablet f750aa46a44f43a7934641028dbb358f
I20260924 12:46:55.573722 10395 tablet_service.cc:1514] Processing DeleteTablet for tablet f750aa46a44f43a7934641028dbb358f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:55 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 12:46:55.573784 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet f750aa46a44f43a7934641028dbb358f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:55 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:53144
I20260924 12:46:55.573985 10255 tablet_service.cc:1514] Processing DeleteTablet for tablet f750aa46a44f43a7934641028dbb358f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:55 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:46270
I20260924 12:46:55.574668 10517 tablet_replica.cc:330] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: stopping tablet replica
I20260924 12:46:55.575179 10517 raft_consensus.cc:2229] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:55.575950 10517 raft_consensus.cc:2258] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:55.575940 9978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:55.576733 9978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:55.582363 10517 ts_tablet_manager.cc:1905] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:55.587227 10254 tablet_service.cc:1467] Processing CreateTablet for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef (DEFAULT_TABLE table=test-table [id=ed2fc6731e9a414ba996b606d0eb8927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:55.587945 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.589553 10394 tablet_service.cc:1467] Processing CreateTablet for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef (DEFAULT_TABLE table=test-table [id=ed2fc6731e9a414ba996b606d0eb8927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:55.590026 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.590207 10115 tablet_service.cc:1467] Processing CreateTablet for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef (DEFAULT_TABLE table=test-table [id=ed2fc6731e9a414ba996b606d0eb8927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:55.590775 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.596987 10490 tablet_bootstrap.cc:492] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Bootstrap starting.
I20260924 12:46:55.598976 10490 tablet_bootstrap.cc:654] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.606921 10517 ts_tablet_manager.cc:1918] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:55.607168 10517 log.cc:1196] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/f750aa46a44f43a7934641028dbb358f
I20260924 12:46:55.607930 10517 ts_tablet_manager.cc:1939] T f750aa46a44f43a7934641028dbb358f P 20173d9e12f948688cbddd800c656289: Deleting consensus metadata
I20260924 12:46:55.610025 9963 catalog_manager.cc:4841] TS 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599): tablet f750aa46a44f43a7934641028dbb358f (table test-table [id=6546ace19d23472397a5f2096fb6e0a4]) successfully deleted
I20260924 12:46:55.612684 10490 tablet_bootstrap.cc:492] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: No bootstrap required, opened a new log
I20260924 12:46:55.612869 10490 ts_tablet_manager.cc:1397] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260924 12:46:55.613729 10490 raft_consensus.cc:348] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.613974 10490 raft_consensus.cc:374] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.614073 10490 raft_consensus.cc:729] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.614395 10490 consensus_queue.cc:260] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.615710 10490 ts_tablet_manager.cc:1428] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 12:46:55.621496 10491 tablet_bootstrap.cc:492] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Bootstrap starting.
I20260924 12:46:55.621899 10489 tablet_bootstrap.cc:492] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Bootstrap starting.
I20260924 12:46:55.622767 10518 tablet_replica.cc:330] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: stopping tablet replica
I20260924 12:46:55.623448 10491 tablet_bootstrap.cc:654] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.623677 10489 tablet_bootstrap.cc:654] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.623238 10518 raft_consensus.cc:2229] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:55.624197 10518 raft_consensus.cc:2258] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:55.625037 10516 tablet_replica.cc:330] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: stopping tablet replica
I20260924 12:46:55.625856 10516 raft_consensus.cc:2229] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:55.626381 10516 raft_consensus.cc:2258] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:55.628094 10518 ts_tablet_manager.cc:1905] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:55.632522 10491 tablet_bootstrap.cc:492] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: No bootstrap required, opened a new log
I20260924 12:46:55.632519 10489 tablet_bootstrap.cc:492] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: No bootstrap required, opened a new log
I20260924 12:46:55.632691 10491 ts_tablet_manager.cc:1397] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 12:46:55.632714 10489 ts_tablet_manager.cc:1397] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260924 12:46:55.633572 10491 raft_consensus.cc:348] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.633569 10489 raft_consensus.cc:348] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.633803 10491 raft_consensus.cc:374] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.633807 10489 raft_consensus.cc:374] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.633911 10491 raft_consensus.cc:729] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20173d9e12f948688cbddd800c656289, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.633919 10489 raft_consensus.cc:729] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b414ee801ab4e198b298a99db0f6af0, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.634212 10489 consensus_queue.cc:260] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.634224 10491 consensus_queue.cc:260] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.634799 10491 ts_tablet_manager.cc:1428] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:55.636907 10516 ts_tablet_manager.cc:1905] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:55.637923 10518 ts_tablet_manager.cc:1918] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:55.638078 10518 log.cc:1196] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/f750aa46a44f43a7934641028dbb358f
I20260924 12:46:55.638571 10518 ts_tablet_manager.cc:1939] T f750aa46a44f43a7934641028dbb358f P 4b414ee801ab4e198b298a99db0f6af0: Deleting consensus metadata
I20260924 12:46:55.639537 10489 ts_tablet_manager.cc:1428] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260924 12:46:55.643515 9966 catalog_manager.cc:4841] TS 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653): tablet f750aa46a44f43a7934641028dbb358f (table test-table [id=6546ace19d23472397a5f2096fb6e0a4]) successfully deleted
I20260924 12:46:55.644012 10516 ts_tablet_manager.cc:1918] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:55.644137 10516 log.cc:1196] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/f750aa46a44f43a7934641028dbb358f
I20260924 12:46:55.644482 10516 ts_tablet_manager.cc:1939] T f750aa46a44f43a7934641028dbb358f P 7f779de13b94419db3fdedba7d249d5e: Deleting consensus metadata
I20260924 12:46:55.646071 9964 catalog_manager.cc:4841] TS 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527): tablet f750aa46a44f43a7934641028dbb358f (table test-table [id=6546ace19d23472397a5f2096fb6e0a4]) successfully deleted
I20260924 12:46:55.749104 10497 raft_consensus.cc:482] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:55.749325 10497 raft_consensus.cc:504] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.750916 10497 leader_election.cc:288] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:55.758882 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.751738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35653 (local address 127.2.81.67:33721)
0924 12:46:55.752437 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.752454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.752592 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.753043 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.753049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.753121 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.753432 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.753445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.754709 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.754714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.756479 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.756493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.758064 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.758070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.758238 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.758271 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.758359 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.758489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":406,"threads_started":1}
I20260924 12:46:55.758845 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.751374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:33721 (local address 127.2.81.66:35653)
0924 12:46:55.751545 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.751551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.752560 (+ 1009us) server_negotiation.cc:407] Connection header received
0924 12:46:55.752751 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.752756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.752884 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.752991 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.753671 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.754577 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.756650 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.757872 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.758008 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.758116 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.758389 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.758525 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:46:55.759591 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30a8c86b4a44d0ca7787e3d4a69a1ef" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0" is_pre_election: true
I20260924 12:46:55.759919 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.751341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:51295 (local address 127.2.81.65:38599)
0924 12:46:55.751562 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.751568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.751996 (+ 428us) server_negotiation.cc:407] Connection header received
0924 12:46:55.752244 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.752249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.752347 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.752400 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:46:55.754205 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.755331 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.757323 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.758255 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.758387 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.758543 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.759450 (+ 907us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.759595 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:46:55.759895 10280 raft_consensus.cc:2454] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 0.
I20260924 12:46:55.760187 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.751229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:38599 (local address 127.2.81.67:51295)
0924 12:46:55.751859 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.751880 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.752053 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.753715 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.753720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.753746 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:46:55.754038 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.754049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.755484 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.755490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.757165 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.757176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.758744 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.758750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.758934 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.759328 (+ 394us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.759454 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.759620 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":337,"threads_started":1}
I20260924 12:46:55.760453 10348 leader_election.cc:302] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b414ee801ab4e198b298a99db0f6af0, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:55.761116 10497 raft_consensus.cc:2781] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:55.761233 10497 raft_consensus.cc:482] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:55.761333 10497 raft_consensus.cc:3037] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.761454 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30a8c86b4a44d0ca7787e3d4a69a1ef" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289" is_pre_election: true
I20260924 12:46:55.761775 10136 raft_consensus.cc:2454] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 0.
I20260924 12:46:55.763630 10497 raft_consensus.cc:504] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.764181 10497 leader_election.cc:288] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 election: Requested vote from peers 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:55.764721 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30a8c86b4a44d0ca7787e3d4a69a1ef" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289"
I20260924 12:46:55.764731 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30a8c86b4a44d0ca7787e3d4a69a1ef" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0"
I20260924 12:46:55.764936 10280 raft_consensus.cc:3037] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.764966 10136 raft_consensus.cc:3037] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.767879 10280 raft_consensus.cc:2454] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 1.
I20260924 12:46:55.768193 10136 raft_consensus.cc:2454] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 1.
I20260924 12:46:55.768590 10348 leader_election.cc:302] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 4b414ee801ab4e198b298a99db0f6af0, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:55.768913 10497 raft_consensus.cc:2781] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:55.769435 10497 raft_consensus.cc:686] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 1 LEADER]: Becoming Leader. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Running, Role: LEADER
I20260924 12:46:55.769836 10497 consensus_queue.cc:237] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.773279 9978 catalog_manager.cc:5495] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f779de13b94419db3fdedba7d249d5e (127.2.81.67). New cstate: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } health_report { overall_health: HEALTHY } } }
Sep 24 12:46:55 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254013, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.126@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:55.853459 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.835228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:35891 (local address 127.0.0.1:52930)
0924 12:46:55.835845 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:55.835866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:55.836019 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:55.836417 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:55.836422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:55.837060 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:46:55.837240 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.837250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.838269 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.838273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.839228 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:55.839238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.839864 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.839870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:55.840011 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.845236 (+ 5225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:46:55.845258 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:46:55.847780 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:46:55.851004 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:55.851018 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:55.851026 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:55.851434 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:55.851777 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:46:55.851782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:46:55.851786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:46:55.851909 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:46:55.852392 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:46:55.852401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:46:55.852550 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 12:46:55.852697 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 12:46:55.853041 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":323,"threads_started":1}
I20260924 12:46:55.853507 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:55.835310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.2.81.126:35891)
0924 12:46:55.836168 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:55.836171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:55.836187 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:46:55.836224 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:55.836227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:55.836279 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:55.836359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:55.837372 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.838139 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.839353 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:55.839717 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:55.839818 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:55.848015 (+ 8197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:55.848020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:55.848023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:55.848053 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:55.850859 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.851541 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.851546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.851550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.851667 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:55.852003 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:55.852008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:55.852012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:55.852272 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:55.852392 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:55.852808 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 12:46:55.853164 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":753,"thread_start_us":592,"threads_started":1}
I20260924 12:46:55.858613 9978 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:52930:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 12:46:55.859733 9978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:55.866565 10115 tablet_service.cc:1467] Processing CreateTablet for tablet d22d2d7ead3a45ce84c52c36b4b79779 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c997ec78eea41068720919719af4e8e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:55.866712 10255 tablet_service.cc:1467] Processing CreateTablet for tablet d22d2d7ead3a45ce84c52c36b4b79779 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c997ec78eea41068720919719af4e8e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:55.867165 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22d2d7ead3a45ce84c52c36b4b79779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.867264 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22d2d7ead3a45ce84c52c36b4b79779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.872792 10489 tablet_bootstrap.cc:492] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: Bootstrap starting.
I20260924 12:46:55.873522 10395 tablet_service.cc:1467] Processing CreateTablet for tablet d22d2d7ead3a45ce84c52c36b4b79779 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c997ec78eea41068720919719af4e8e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 12:46:55.874073 10489 tablet_bootstrap.cc:654] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.874070 10395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22d2d7ead3a45ce84c52c36b4b79779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:55.875948 10489 tablet_bootstrap.cc:492] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: No bootstrap required, opened a new log
I20260924 12:46:55.876094 10489 ts_tablet_manager.cc:1397] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:55.876780 10489 raft_consensus.cc:348] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.876950 10489 raft_consensus.cc:374] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.877085 10489 raft_consensus.cc:729] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b414ee801ab4e198b298a99db0f6af0, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.877332 10489 consensus_queue.cc:260] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.877899 10489 ts_tablet_manager.cc:1428] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:55.877787 10495 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:55.878064 10495 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:55.882375 10490 tablet_bootstrap.cc:492] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: Bootstrap starting.
I20260924 12:46:55.883631 10490 tablet_bootstrap.cc:654] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.884169 10491 tablet_bootstrap.cc:492] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: Bootstrap starting.
I20260924 12:46:55.885984 10491 tablet_bootstrap.cc:654] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:55.886782 10490 tablet_bootstrap.cc:492] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: No bootstrap required, opened a new log
I20260924 12:46:55.886955 10490 ts_tablet_manager.cc:1397] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 12:46:55.887774 10490 raft_consensus.cc:348] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.888038 10490 raft_consensus.cc:374] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.888114 10490 raft_consensus.cc:729] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.888358 10490 consensus_queue.cc:260] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.889079 10490 ts_tablet_manager.cc:1428] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 12:46:55.888893 10497 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:55.889782 10497 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:55.890717 10491 tablet_bootstrap.cc:492] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: No bootstrap required, opened a new log
I20260924 12:46:55.890883 10491 ts_tablet_manager.cc:1397] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 12:46:55.891467 10491 raft_consensus.cc:348] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.891657 10491 raft_consensus.cc:374] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:55.891739 10491 raft_consensus.cc:729] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20173d9e12f948688cbddd800c656289, State: Initialized, Role: FOLLOWER
I20260924 12:46:55.891994 10491 consensus_queue.cc:260] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.892482 10491 ts_tablet_manager.cc:1428] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 12:46:55.892504 10511 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:55.892875 10511 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:55.985030 10497 raft_consensus.cc:482] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:55.985261 10497 raft_consensus.cc:504] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.985919 10497 leader_election.cc:288] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:55.986549 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22d2d7ead3a45ce84c52c36b4b79779" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289" is_pre_election: true
I20260924 12:46:55.986603 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22d2d7ead3a45ce84c52c36b4b79779" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0" is_pre_election: true
I20260924 12:46:55.986793 10136 raft_consensus.cc:2454] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 0.
I20260924 12:46:55.986819 10280 raft_consensus.cc:2454] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 0.
I20260924 12:46:55.987283 10348 leader_election.cc:302] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b414ee801ab4e198b298a99db0f6af0, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:55.987623 10497 raft_consensus.cc:2781] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:55.987736 10497 raft_consensus.cc:482] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:55.987834 10497 raft_consensus.cc:3037] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.990679 10497 raft_consensus.cc:504] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.991345 10497 leader_election.cc:288] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 election: Requested vote from peers 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:55.991858 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22d2d7ead3a45ce84c52c36b4b79779" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0"
I20260924 12:46:55.991915 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22d2d7ead3a45ce84c52c36b4b79779" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289"
I20260924 12:46:55.992084 10280 raft_consensus.cc:3037] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.992108 10136 raft_consensus.cc:3037] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:55.994697 10136 raft_consensus.cc:2454] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 1.
I20260924 12:46:55.995040 10280 raft_consensus.cc:2454] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 1.
I20260924 12:46:55.995134 10345 leader_election.cc:302] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:55.995465 10497 raft_consensus.cc:2781] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:55.995637 10497 raft_consensus.cc:686] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [term 1 LEADER]: Becoming Leader. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Running, Role: LEADER
I20260924 12:46:55.995883 10497 consensus_queue.cc:237] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:55.997427 10524 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: TxnStatusTablet state changed. Reason: New leader 7f779de13b94419db3fdedba7d249d5e. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:55.997763 10524 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:55.998360 10537 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 12:46:55.998567 10537 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 12:46:55.999014 9979 catalog_manager.cc:5495] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f779de13b94419db3fdedba7d249d5e (127.2.81.67). New cstate: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } health_report { overall_health: HEALTHY } } }
I20260924 12:46:56.038793 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.032759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41527 (local address 127.0.0.1:33634)
0924 12:46:56.033188 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:56.033208 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:56.033358 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:56.034043 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:56.034048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:56.034060 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:56.034244 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.034253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.035329 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.035334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.036321 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.036333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.037063 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.037070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.037235 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.037293 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.038231 (+ 938us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.038253 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 12:46:56.038332 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 12:46:56.038492 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260924 12:46:56.038852 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.032772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.2.81.67:41527)
0924 12:46:56.033699 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.033703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.033721 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:56.033793 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.033800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.033869 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.034003 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.034400 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.035206 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.036463 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.036883 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.037021 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.037517 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.038119 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.038254 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.038382 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.038474 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":635,"threads_started":1}
I20260924 12:46:56.042876 10280 raft_consensus.cc:1261] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Refusing update from remote peer 7f779de13b94419db3fdedba7d249d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.043123 10136 raft_consensus.cc:1261] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Refusing update from remote peer 7f779de13b94419db3fdedba7d249d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.043666 10524 consensus_queue.cc:1035] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.044080 10497 consensus_queue.cc:1035] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.057498 10495 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: TxnStatusTablet state changed. Reason: New leader 7f779de13b94419db3fdedba7d249d5e. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:56.057770 10495 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:56.057845 10511 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: TxnStatusTablet state changed. Reason: New leader 7f779de13b94419db3fdedba7d249d5e. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:56.058127 10511 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:56.060056 10497 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:56.060453 10497 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:56.063851 10497 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:56.067328 10511 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:56.067623 10511 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 20173d9e12f948688cbddd800c656289: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:56.068950 10497 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 7f779de13b94419db3fdedba7d249d5e: This TxnStatusTablet replica's current role is: LEADER
I20260924 12:46:56.072588 10495 tablet_replica.cc:439] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } } }
I20260924 12:46:56.072871 10495 tablet_replica.cc:442] T d22d2d7ead3a45ce84c52c36b4b79779 P 4b414ee801ab4e198b298a99db0f6af0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 12:46:56.077455 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.069241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.2.81.67:41527)
0924 12:46:56.069429 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.069435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.071133 (+ 1698us) server_negotiation.cc:407] Connection header received
0924 12:46:56.071402 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.071407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.071494 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.071565 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.072263 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.073399 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.075291 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.075952 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.076059 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.076657 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.076761 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.076856 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.077060 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.077141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:46:56.093856 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.087941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:45749 (local address 127.2.81.67:41527)
0924 12:46:56.088136 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.088142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.088804 (+ 662us) server_negotiation.cc:407] Connection header received
0924 12:46:56.089019 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.089024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.089103 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.089173 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.089671 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.090596 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.092072 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.092518 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.092627 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.093090 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.093198 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.093270 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.093399 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.093465 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 12:46:56.094353 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.087975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41527 (local address 127.2.81.67:45749)
0924 12:46:56.088707 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:56.088723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:56.088877 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:56.089196 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:56.089201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:56.089210 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:56.089537 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.089546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.090711 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.090717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.091940 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.091953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.092711 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.092718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.092841 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.092896 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.093294 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.093304 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 12:46:56.093374 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 12:46:56.093509 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":437,"threads_started":1}
I20260924 12:46:56.103909 10280 raft_consensus.cc:1261] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Refusing update from remote peer 7f779de13b94419db3fdedba7d249d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.103952 10136 raft_consensus.cc:1261] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Refusing update from remote peer 7f779de13b94419db3fdedba7d249d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.104451 10497 consensus_queue.cc:1035] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.104732 10524 consensus_queue.cc:1035] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.114324 10508 mvcc.cc:205] Tried to move back new op lower bound from 7332880449956683776 to 7332880448596832256. Current Snapshot: MvccSnapshot[applied={T|T < 7332880449956683776}]
I20260924 12:46:56.172958 9977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:56.173250 9977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:56.177062 9977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Sending DeleteTablet for 3 replicas of tablet d30a8c86b4a44d0ca7787e3d4a69a1ef
I20260924 12:46:56.177875 10115 tablet_service.cc:1514] Processing DeleteTablet for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:53144
I20260924 12:46:56.178102 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 12:46:56.178320 10255 tablet_service.cc:1514] Processing DeleteTablet for tablet d30a8c86b4a44d0ca7787e3d4a69a1ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:46270
I20260924 12:46:56.179534 10556 tablet_replica.cc:330] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: stopping tablet replica
I20260924 12:46:56.179745 10556 raft_consensus.cc:2229] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:56.179948 10557 tablet_replica.cc:330] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: stopping tablet replica
I20260924 12:46:56.180112 10557 raft_consensus.cc:2229] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:56.180315 10557 raft_consensus.cc:2258] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.180310 10556 raft_consensus.cc:2258] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.181399 10557 ts_tablet_manager.cc:1905] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.181411 10556 ts_tablet_manager.cc:1905] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.188055 10557 ts_tablet_manager.cc:1918] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:56.188251 10557 log.cc:1196] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/d30a8c86b4a44d0ca7787e3d4a69a1ef
I20260924 12:46:56.188642 10556 ts_tablet_manager.cc:1918] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:56.188774 10557 ts_tablet_manager.cc:1939] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 4b414ee801ab4e198b298a99db0f6af0: Deleting consensus metadata
I20260924 12:46:56.188786 10556 log.cc:1196] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/d30a8c86b4a44d0ca7787e3d4a69a1ef
I20260924 12:46:56.189260 10556 ts_tablet_manager.cc:1939] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 7f779de13b94419db3fdedba7d249d5e: Deleting consensus metadata
I20260924 12:46:56.190080 9966 catalog_manager.cc:4841] TS 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653): tablet d30a8c86b4a44d0ca7787e3d4a69a1ef (table test-table [id=ed2fc6731e9a414ba996b606d0eb8927]) successfully deleted
I20260924 12:46:56.190382 9964 catalog_manager.cc:4841] TS 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527): tablet d30a8c86b4a44d0ca7787e3d4a69a1ef (table test-table [id=ed2fc6731e9a414ba996b606d0eb8927]) successfully deleted
I20260924 12:46:56.191017 10555 tablet_replica.cc:330] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: stopping tablet replica
I20260924 12:46:56.191224 10555 raft_consensus.cc:2229] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:56.191448 10555 raft_consensus.cc:2258] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.192636 10555 ts_tablet_manager.cc:1905] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.200661 10555 ts_tablet_manager.cc:1918] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:56.200821 10555 log.cc:1196] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/d30a8c86b4a44d0ca7787e3d4a69a1ef
I20260924 12:46:56.201283 10555 ts_tablet_manager.cc:1939] T d30a8c86b4a44d0ca7787e3d4a69a1ef P 20173d9e12f948688cbddd800c656289: Deleting consensus metadata
I20260924 12:46:56.202509 9963 catalog_manager.cc:4841] TS 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599): tablet d30a8c86b4a44d0ca7787e3d4a69a1ef (table test-table [id=ed2fc6731e9a414ba996b606d0eb8927]) successfully deleted
Sep 24 12:46:56 dist-test-slave-9tm1 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 12:46:56 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:46:56 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 12:46:56.238201 10532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:46:56.226715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.2.81.126:35891)
0924 12:46:56.227006 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.227013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.227255 (+ 242us) server_negotiation.cc:407] Connection header received
0924 12:46:56.227449 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.227453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.227526 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.227618 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:56.229284 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.230031 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.231826 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.232268 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.232368 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.237939 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52940: BlockingRecv error: recv got EOF from 127.0.0.1:52940 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 12:46:56 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM
I20260924 12:46:56.256808 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.240964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.2.81.126:35891)
0924 12:46:56.241204 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.241211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.241685 (+ 474us) server_negotiation.cc:407] Connection header received
0924 12:46:56.241848 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.241852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.241925 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.242008 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:56.243153 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.244268 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.245589 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.246217 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.246344 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.250332 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:56.250339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:56.250345 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:56.250391 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:56.253885 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:56.254657 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:56.254663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:56.254667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:56.254841 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:56.255257 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:56.255263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:56.255267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:56.255611 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:56.255867 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.256250 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.256478 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:46:56.260658 9977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:46:56.261376 9977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:56.268167 10394 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5c32ad876f4889bfb6d22011afb9e8 (DEFAULT_TABLE table=test-table [id=e557affa272f46ba88e310cb53196fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:56.268164 10115 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5c32ad876f4889bfb6d22011afb9e8 (DEFAULT_TABLE table=test-table [id=e557affa272f46ba88e310cb53196fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:56.268669 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5c32ad876f4889bfb6d22011afb9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:56.268673 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5c32ad876f4889bfb6d22011afb9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:56.275454 10490 tablet_bootstrap.cc:492] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Bootstrap starting.
I20260924 12:46:56.277173 10255 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5c32ad876f4889bfb6d22011afb9e8 (DEFAULT_TABLE table=test-table [id=e557affa272f46ba88e310cb53196fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:56.277594 10490 tablet_bootstrap.cc:654] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:56.277652 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5c32ad876f4889bfb6d22011afb9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:56.279529 10490 tablet_bootstrap.cc:492] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: No bootstrap required, opened a new log
I20260924 12:46:56.279654 10490 ts_tablet_manager.cc:1397] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 12:46:56.280294 10490 raft_consensus.cc:348] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.280494 10490 raft_consensus.cc:374] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:56.280567 10490 raft_consensus.cc:729] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Initialized, Role: FOLLOWER
I20260924 12:46:56.280764 10490 consensus_queue.cc:260] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.281332 10490 ts_tablet_manager.cc:1428] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:56.285218 10489 tablet_bootstrap.cc:492] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Bootstrap starting.
I20260924 12:46:56.285737 10491 tablet_bootstrap.cc:492] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Bootstrap starting.
I20260924 12:46:56.286810 10489 tablet_bootstrap.cc:654] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:56.287217 10491 tablet_bootstrap.cc:654] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:56.289067 10489 tablet_bootstrap.cc:492] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: No bootstrap required, opened a new log
I20260924 12:46:56.289222 10489 ts_tablet_manager.cc:1397] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 12:46:56.289810 10489 raft_consensus.cc:348] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.290063 10489 raft_consensus.cc:374] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:56.290151 10489 raft_consensus.cc:729] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b414ee801ab4e198b298a99db0f6af0, State: Initialized, Role: FOLLOWER
I20260924 12:46:56.290658 10491 tablet_bootstrap.cc:492] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: No bootstrap required, opened a new log
I20260924 12:46:56.290369 10489 consensus_queue.cc:260] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.290812 10491 ts_tablet_manager.cc:1397] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 12:46:56.291183 10489 ts_tablet_manager.cc:1428] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:56.291708 10491 raft_consensus.cc:348] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.291985 10491 raft_consensus.cc:374] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:56.292081 10491 raft_consensus.cc:729] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20173d9e12f948688cbddd800c656289, State: Initialized, Role: FOLLOWER
I20260924 12:46:56.292331 10491 consensus_queue.cc:260] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.293684 10491 ts_tablet_manager.cc:1428] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:56.380456 10524 raft_consensus.cc:482] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:56.380703 10524 raft_consensus.cc:504] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.381608 10524 leader_election.cc:288] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:56.382196 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c32ad876f4889bfb6d22011afb9e8" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0" is_pre_election: true
I20260924 12:46:56.382210 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c32ad876f4889bfb6d22011afb9e8" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289" is_pre_election: true
I20260924 12:46:56.382423 10280 raft_consensus.cc:2454] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 0.
I20260924 12:46:56.382471 10136 raft_consensus.cc:2454] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 0.
I20260924 12:46:56.382831 10348 leader_election.cc:302] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b414ee801ab4e198b298a99db0f6af0, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:56.383124 10524 raft_consensus.cc:2781] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:56.383265 10524 raft_consensus.cc:482] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:56.383373 10524 raft_consensus.cc:3037] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:56.386032 10524 raft_consensus.cc:504] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.386751 10524 leader_election.cc:288] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 election: Requested vote from peers 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599), 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653)
I20260924 12:46:56.387315 10136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c32ad876f4889bfb6d22011afb9e8" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20173d9e12f948688cbddd800c656289"
I20260924 12:46:56.387356 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5c32ad876f4889bfb6d22011afb9e8" candidate_uuid: "7f779de13b94419db3fdedba7d249d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0"
I20260924 12:46:56.387550 10136 raft_consensus.cc:3037] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:56.387573 10280 raft_consensus.cc:3037] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:56.389653 10136 raft_consensus.cc:2454] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 1.
I20260924 12:46:56.390134 10345 leader_election.cc:302] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:56.390504 10524 raft_consensus.cc:2781] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:56.390661 10524 raft_consensus.cc:686] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 1 LEADER]: Becoming Leader. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Running, Role: LEADER
I20260924 12:46:56.390643 10280 raft_consensus.cc:2454] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f779de13b94419db3fdedba7d249d5e in term 1.
I20260924 12:46:56.390957 10524 consensus_queue.cc:237] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.393822 9978 catalog_manager.cc:5495] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f779de13b94419db3fdedba7d249d5e (127.2.81.67). New cstate: current_term: 1 leader_uuid: "7f779de13b94419db3fdedba7d249d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } health_report { overall_health: HEALTHY } } }
I20260924 12:46:56.411048 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.406347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.2.81.67:41527)
0924 12:46:56.406541 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.406547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.406566 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:46:56.406659 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.406664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.406732 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.406810 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.407160 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.407934 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.409287 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.409780 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.409896 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.410331 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.410463 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.410545 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.410684 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.410762 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 12:46:56.414021 10280 raft_consensus.cc:1261] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Refusing update from remote peer 7f779de13b94419db3fdedba7d249d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.414199 10136 raft_consensus.cc:1261] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Refusing update from remote peer 7f779de13b94419db3fdedba7d249d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.414732 10524 consensus_queue.cc:1035] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.415025 10497 consensus_queue.cc:1035] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.428817 9979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:56.429064 9979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:56.431989 9979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Sending DeleteTablet for 3 replicas of tablet 4b5c32ad876f4889bfb6d22011afb9e8
I20260924 12:46:56.432849 10115 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5c32ad876f4889bfb6d22011afb9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:53144
I20260924 12:46:56.433221 10255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5c32ad876f4889bfb6d22011afb9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:46270
I20260924 12:46:56.433305 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5c32ad876f4889bfb6d22011afb9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 12:46:56.433498 10555 tablet_replica.cc:330] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: stopping tablet replica
I20260924 12:46:56.433688 10555 raft_consensus.cc:2229] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:56.433939 10555 raft_consensus.cc:2258] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.434216 10557 tablet_replica.cc:330] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: stopping tablet replica
I20260924 12:46:56.434096 9979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 12:46:56.434396 10557 raft_consensus.cc:2229] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:56.434576 10557 raft_consensus.cc:2258] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 12:46:56.434701 9979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:46:56.434824 10556 tablet_replica.cc:330] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: stopping tablet replica
I20260924 12:46:56.435005 10556 raft_consensus.cc:2229] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:56.435326 10556 raft_consensus.cc:2258] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.436431 10557 ts_tablet_manager.cc:1905] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.436616 10555 ts_tablet_manager.cc:1905] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.437763 10556 ts_tablet_manager.cc:1905] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.442405 10394 tablet_service.cc:1467] Processing CreateTablet for tablet 19d42027636641f9aebccf0fba9e34f4 (DEFAULT_TABLE table=test-table [id=18835ebe86874b8fa33add618f812878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:56.444324 10557 ts_tablet_manager.cc:1918] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:56.444473 10557 log.cc:1196] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/4b5c32ad876f4889bfb6d22011afb9e8
I20260924 12:46:56.444782 10557 ts_tablet_manager.cc:1939] T 4b5c32ad876f4889bfb6d22011afb9e8 P 4b414ee801ab4e198b298a99db0f6af0: Deleting consensus metadata
I20260924 12:46:56.445596 10255 tablet_service.cc:1467] Processing CreateTablet for tablet 19d42027636641f9aebccf0fba9e34f4 (DEFAULT_TABLE table=test-table [id=18835ebe86874b8fa33add618f812878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:56.445806 9966 catalog_manager.cc:4841] TS 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653): tablet 4b5c32ad876f4889bfb6d22011afb9e8 (table test-table [id=e557affa272f46ba88e310cb53196fa4]) successfully deleted
I20260924 12:46:56.446028 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d42027636641f9aebccf0fba9e34f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:56.449402 10115 tablet_service.cc:1467] Processing CreateTablet for tablet 19d42027636641f9aebccf0fba9e34f4 (DEFAULT_TABLE table=test-table [id=18835ebe86874b8fa33add618f812878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:46:56.449795 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d42027636641f9aebccf0fba9e34f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:56.453015 10489 tablet_bootstrap.cc:492] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Bootstrap starting.
I20260924 12:46:56.454654 10489 tablet_bootstrap.cc:654] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:56.455917 10491 tablet_bootstrap.cc:492] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Bootstrap starting.
I20260924 12:46:56.456060 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d42027636641f9aebccf0fba9e34f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:46:56.457517 10491 tablet_bootstrap.cc:654] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:56.465183 10491 tablet_bootstrap.cc:492] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: No bootstrap required, opened a new log
I20260924 12:46:56.465505 10491 ts_tablet_manager.cc:1397] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 12:46:56.466293 10555 ts_tablet_manager.cc:1918] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:56.466461 10555 log.cc:1196] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/4b5c32ad876f4889bfb6d22011afb9e8
I20260924 12:46:56.466624 10491 raft_consensus.cc:348] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.466761 10556 ts_tablet_manager.cc:1918] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:46:56.466929 10555 ts_tablet_manager.cc:1939] T 4b5c32ad876f4889bfb6d22011afb9e8 P 20173d9e12f948688cbddd800c656289: Deleting consensus metadata
I20260924 12:46:56.467032 10491 raft_consensus.cc:374] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:56.467154 10491 raft_consensus.cc:729] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20173d9e12f948688cbddd800c656289, State: Initialized, Role: FOLLOWER
I20260924 12:46:56.467476 10491 consensus_queue.cc:260] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.467972 9963 catalog_manager.cc:4841] TS 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599): tablet 4b5c32ad876f4889bfb6d22011afb9e8 (table test-table [id=e557affa272f46ba88e310cb53196fa4]) successfully deleted
I20260924 12:46:56.468843 10491 ts_tablet_manager.cc:1428] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 12:46:56.470952 10489 tablet_bootstrap.cc:492] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: No bootstrap required, opened a new log
I20260924 12:46:56.471118 10489 ts_tablet_manager.cc:1397] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20260924 12:46:56.471740 10489 raft_consensus.cc:348] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.472008 10489 raft_consensus.cc:374] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:56.472084 10489 raft_consensus.cc:729] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b414ee801ab4e198b298a99db0f6af0, State: Initialized, Role: FOLLOWER
I20260924 12:46:56.472363 10489 consensus_queue.cc:260] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.473127 10556 log.cc:1196] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/4b5c32ad876f4889bfb6d22011afb9e8
I20260924 12:46:56.473774 10556 ts_tablet_manager.cc:1939] T 4b5c32ad876f4889bfb6d22011afb9e8 P 7f779de13b94419db3fdedba7d249d5e: Deleting consensus metadata
I20260924 12:46:56.474004 10489 ts_tablet_manager.cc:1428] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 12:46:56.475298 10490 tablet_bootstrap.cc:492] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Bootstrap starting.
I20260924 12:46:56.476823 9964 catalog_manager.cc:4841] TS 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527): tablet 4b5c32ad876f4889bfb6d22011afb9e8 (table test-table [id=e557affa272f46ba88e310cb53196fa4]) successfully deleted
I20260924 12:46:56.477123 10490 tablet_bootstrap.cc:654] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Neither blocks nor log segments found. Creating new log.
I20260924 12:46:56.479681 10490 tablet_bootstrap.cc:492] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: No bootstrap required, opened a new log
I20260924 12:46:56.479876 10490 ts_tablet_manager.cc:1397] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 12:46:56.480666 10490 raft_consensus.cc:348] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.480937 10490 raft_consensus.cc:374] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:46:56.481053 10490 raft_consensus.cc:729] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f779de13b94419db3fdedba7d249d5e, State: Initialized, Role: FOLLOWER
I20260924 12:46:56.481302 10490 consensus_queue.cc:260] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.481874 10490 ts_tablet_manager.cc:1428] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 12:46:56.538496 10511 raft_consensus.cc:482] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:46:56.538688 10511 raft_consensus.cc:504] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.539423 10511 leader_election.cc:288] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653), 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527)
I20260924 12:46:56.539956 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d42027636641f9aebccf0fba9e34f4" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0" is_pre_election: true
I20260924 12:46:56.540078 10415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d42027636641f9aebccf0fba9e34f4" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f779de13b94419db3fdedba7d249d5e" is_pre_election: true
I20260924 12:46:56.540253 10280 raft_consensus.cc:2454] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 0.
I20260924 12:46:56.540313 10415 raft_consensus.cc:2454] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 0.
I20260924 12:46:56.540778 10067 leader_election.cc:302] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:56.541115 10511 raft_consensus.cc:2781] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:46:56.541234 10511 raft_consensus.cc:482] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:46:56.541320 10511 raft_consensus.cc:3037] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:56.543354 10511 raft_consensus.cc:504] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.543949 10511 leader_election.cc:288] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 election: Requested vote from peers 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653), 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527)
I20260924 12:46:56.544466 10415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d42027636641f9aebccf0fba9e34f4" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f779de13b94419db3fdedba7d249d5e"
I20260924 12:46:56.544474 10280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d42027636641f9aebccf0fba9e34f4" candidate_uuid: "20173d9e12f948688cbddd800c656289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b414ee801ab4e198b298a99db0f6af0"
I20260924 12:46:56.544660 10415 raft_consensus.cc:3037] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:56.544709 10280 raft_consensus.cc:3037] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:46:56.546816 10415 raft_consensus.cc:2454] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 1.
I20260924 12:46:56.547254 10067 leader_election.cc:302] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20173d9e12f948688cbddd800c656289, 7f779de13b94419db3fdedba7d249d5e; no voters:
I20260924 12:46:56.547573 10511 raft_consensus.cc:2781] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:46:56.547652 10280 raft_consensus.cc:2454] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20173d9e12f948688cbddd800c656289 in term 1.
I20260924 12:46:56.558691 10511 raft_consensus.cc:686] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 1 LEADER]: Becoming Leader. State: Replica: 20173d9e12f948688cbddd800c656289, State: Running, Role: LEADER
I20260924 12:46:56.559458 10511 consensus_queue.cc:237] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } }
I20260924 12:46:56.563481 9979 catalog_manager.cc:5495] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20173d9e12f948688cbddd800c656289 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "20173d9e12f948688cbddd800c656289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20173d9e12f948688cbddd800c656289" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 38599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 } health_report { overall_health: UNKNOWN } } }
I20260924 12:46:56.592185 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.587109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.2.81.65:38599)
0924 12:46:56.587644 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.587650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.587667 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:46:56.587730 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.587735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.587811 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.587904 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.588249 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.589103 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.590409 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.590826 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.590931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.591406 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.591516 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.591588 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.591758 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.591842 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":258,"threads_started":1}
I20260924 12:46:56.611223 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.603644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:52349 (local address 127.2.81.67:41527)
0924 12:46:56.603854 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.603861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.605735 (+ 1874us) server_negotiation.cc:407] Connection header received
0924 12:46:56.605900 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.605906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.606000 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.606115 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.606513 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.607648 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.609073 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.609783 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.609918 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.610382 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.610561 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.610653 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.610809 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.610896 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:46:56.611254 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.603403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41527 (local address 127.2.81.65:52349)
0924 12:46:56.605617 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:56.605634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:56.605770 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:56.606124 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:56.606127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:56.606135 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:56.606368 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.606375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.607797 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.607801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.608877 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.608890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.609993 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.610000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.610142 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.610212 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.610680 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.610691 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 12:46:56.610770 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 12:46:56.610918 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1915,"threads_started":1}
I20260924 12:46:56.620069 10415 raft_consensus.cc:1261] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Refusing update from remote peer 20173d9e12f948688cbddd800c656289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.620110 10280 raft_consensus.cc:1261] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Refusing update from remote peer 20173d9e12f948688cbddd800c656289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:46:56.620692 10580 consensus_queue.cc:1035] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b414ee801ab4e198b298a99db0f6af0" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.621037 10511 consensus_queue.cc:1035] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f779de13b94419db3fdedba7d249d5e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:46:56.662336 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.655597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:54011 (local address 127.2.81.65:38599)
0924 12:46:56.655787 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.655793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.656200 (+ 407us) server_negotiation.cc:407] Connection header received
0924 12:46:56.656349 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.656355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.656437 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.656536 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:46:56.657043 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.658023 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.660309 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.660795 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.660944 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.661455 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.661594 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.661685 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.661924 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.662000 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:46:56.662449 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.655539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:38599 (local address 127.2.81.67:54011)
0924 12:46:56.656097 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 12:46:56.656112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:46:56.656243 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:46:56.656580 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:46:56.656588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:46:56.656600 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 12:46:56.656881 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.656892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.658168 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.658174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.660145 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:46:56.660164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.661033 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.661041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:46:56.661215 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.661267 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 12:46:56.661702 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 12:46:56.661715 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 12:46:56.661903 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 12:46:56.662051 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":326,"threads_started":1}
I20260924 12:46:56.689448 9979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:56.689662 9979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:46:56.692782 9979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 477b3ea554d8492c86025c49d8a42311: Sending DeleteTablet for 3 replicas of tablet 19d42027636641f9aebccf0fba9e34f4
I20260924 12:46:56.693732 10115 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d42027636641f9aebccf0fba9e34f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:53144
I20260924 12:46:56.693856 10255 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d42027636641f9aebccf0fba9e34f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:46270
I20260924 12:46:56.694043 10555 tablet_replica.cc:330] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: stopping tablet replica
I20260924 12:46:56.694255 10557 tablet_replica.cc:330] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: stopping tablet replica
I20260924 12:46:56.694341 10555 raft_consensus.cc:2229] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:46:56.694357 10394 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d42027636641f9aebccf0fba9e34f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:46:56 UTC) from {username='oryx', principal='oryx/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 12:46:56.694424 10557 raft_consensus.cc:2229] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:56.694604 10557 raft_consensus.cc:2258] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.694646 10556 tablet_replica.cc:330] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: stopping tablet replica
I20260924 12:46:56.694700 10555 raft_consensus.cc:2258] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.694818 10556 raft_consensus.cc:2229] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:46:56.695009 10556 raft_consensus.cc:2258] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:46:56.695627 10557 ts_tablet_manager.cc:1905] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.696177 10556 ts_tablet_manager.cc:1905] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.696180 10555 ts_tablet_manager.cc:1905] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:46:56.701921 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 10050
I20260924 12:46:56.702601 10555 ts_tablet_manager.cc:1918] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:56.702723 10555 log.cc:1196] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/19d42027636641f9aebccf0fba9e34f4
I20260924 12:46:56.702744 10556 ts_tablet_manager.cc:1918] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:56.702857 10556 log.cc:1196] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/19d42027636641f9aebccf0fba9e34f4
I20260924 12:46:56.703153 10555 ts_tablet_manager.cc:1939] T 19d42027636641f9aebccf0fba9e34f4 P 20173d9e12f948688cbddd800c656289: Deleting consensus metadata
I20260924 12:46:56.703291 10556 ts_tablet_manager.cc:1939] T 19d42027636641f9aebccf0fba9e34f4 P 7f779de13b94419db3fdedba7d249d5e: Deleting consensus metadata
I20260924 12:46:56.704368 9964 catalog_manager.cc:4841] TS 7f779de13b94419db3fdedba7d249d5e (127.2.81.67:41527): tablet 19d42027636641f9aebccf0fba9e34f4 (table test-table [id=18835ebe86874b8fa33add618f812878]) successfully deleted
I20260924 12:46:56.704866 9963 catalog_manager.cc:4841] TS 20173d9e12f948688cbddd800c656289 (127.2.81.65:38599): tablet 19d42027636641f9aebccf0fba9e34f4 (table test-table [id=18835ebe86874b8fa33add618f812878]) successfully deleted
I20260924 12:46:56.709179 10557 ts_tablet_manager.cc:1918] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 12:46:56.709342 10557 log.cc:1196] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/19d42027636641f9aebccf0fba9e34f4
I20260924 12:46:56.709874 10557 ts_tablet_manager.cc:1939] T 19d42027636641f9aebccf0fba9e34f4 P 4b414ee801ab4e198b298a99db0f6af0: Deleting consensus metadata
I20260924 12:46:56.711109 9966 catalog_manager.cc:4841] TS 4b414ee801ab4e198b298a99db0f6af0 (127.2.81.66:35653): tablet 19d42027636641f9aebccf0fba9e34f4 (table test-table [id=18835ebe86874b8fa33add618f812878]) successfully deleted
Sep 24 12:46:56 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.65@KRBTEST.COM
I20260924 12:46:56.723331 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.703618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.2.81.65:38599)
0924 12:46:56.703788 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.703794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.709888 (+ 6094us) server_negotiation.cc:407] Connection header received
0924 12:46:56.710094 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.710099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.710209 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.710301 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:56.711603 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.712342 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.713578 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.714004 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.714112 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.717753 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:56.717766 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:56.717782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:56.717821 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:56.720575 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:56.721313 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:56.721320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:56.721322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:56.721459 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:56.721812 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:56.721816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:56.721818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:56.722176 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:56.722450 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.722775 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.723012 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:46:56.723960 10181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:56.906877 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 10050
I20260924 12:46:56.933413 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 10190
Sep 24 12:46:56 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.66@KRBTEST.COM
I20260924 12:46:56.953181 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:56.934306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.2.81.66:35653)
0924 12:46:56.934781 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:56.934785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:56.934803 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:46:56.934857 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:56.934860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:56.934919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:56.935010 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:56.936758 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.937680 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.939638 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:56.940510 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:56.940712 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:56.946456 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:56.946483 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:56.946505 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:56.946572 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:56.950227 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:56.950975 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:56.950985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:56.950989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:56.951144 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:56.951469 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:56.951473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:56.951477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:56.951927 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:56.952200 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:56.952506 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 12:46:56.952739 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":252,"threads_started":1}
I20260924 12:46:56.953884 10320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:57.053737 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 10190
I20260924 12:46:57.068483 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 10329
Sep 24 12:46:57 dist-test-slave-9tm1 krb5kdc[9934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.81.67@KRBTEST.COM
I20260924 12:46:57.085549 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:46:57.069287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.2.81.67:41527)
0924 12:46:57.069460 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 12:46:57.069465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:46:57.069484 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:46:57.069678 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:46:57.069683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:46:57.069759 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:46:57.069851 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:46:57.071376 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:57.072074 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:57.073560 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:46:57.074090 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:46:57.074210 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:46:57.078701 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:46:57.078722 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:46:57.078743 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:46:57.078790 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:46:57.082292 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:57.083192 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:57.083200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:57.083203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:57.083374 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:46:57.084025 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:46:57.084030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:46:57.084032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:46:57.084428 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:46:57.084677 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 12:46:57.085104 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 12:46:57.085296 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:46:57.086347 10460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:57.233285 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 10329
I20260924 12:46:57.253399 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 9947
I20260924 12:46:57.254652 10019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:46:57.373703 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 9947
2026-09-24T12:46:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5765 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:57 dist-test-slave-9tm1 krb5kdc[10602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:57 dist-test-slave-9tm1 krb5kdc[10602](info): set up 2 sockets
Sep 24 12:46:57 dist-test-slave-9tm1 krb5kdc[10602](info): commencing operation
krb5kdc: starting...
W20260924 12:46:58.414367 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:46:58 dist-test-slave-9tm1 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:46:58.641382 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41413
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:46:58.968703 10615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:46:58.969053 10615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:46:58.969188 10615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:46:58.976622 10615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:46:58.976742 10615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:46:58.976786 10615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:46:58.976840 10615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:46:58.976884 10615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:46:58.987310 10615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:46:58.988583 10615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:46:58.989929 10615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:46:59.002481 10620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:59.002544 10621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:46:59.002537 10623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:46:59.003192 10615 server_base.cc:1034] running on GCE node
I20260924 12:46:59.004016 10615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:46:59.005633 10615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:46:59.006893 10615 hybrid_clock.cc:648] HybridClock initialized: now 1790254019006836 us; error 49 us; skew 500 ppm
I20260924 12:46:59.007294 10615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T12:46:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1696 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:46:59 dist-test-slave-9tm1 krb5kdc[10629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:46:59 dist-test-slave-9tm1 krb5kdc[10629](info): set up 2 sockets
Sep 24 12:46:59 dist-test-slave-9tm1 krb5kdc[10629](info): commencing operation
krb5kdc: starting...
W20260924 12:47:00.121953 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:47:00 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:47:00.350466 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45011
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:00.686614 10642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:00.686936 10642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:00.687065 10642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:00.694975 10642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:47:00.695119 10642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:00.695176 10642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:00.695266 10642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:47:00.695324 10642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:47:00.706100 10642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:00.707444 10642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:00.708709 10642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:00.720543 10647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:00.720520 10648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:00.720873 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:00.721359 10642 server_base.cc:1034] running on GCE node
I20260924 12:47:00.722141 10642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:00.723616 10642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:00.724884 10642 hybrid_clock.cc:648] HybridClock initialized: now 1790254020724791 us; error 81 us; skew 500 ppm
I20260924 12:47:00.725358 10642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:00 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254020, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:00.728691 10642 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:47:00.730422 10642 webserver.cc:460] Webserver started at http://127.2.81.126:39493/ using document root <none> and password file <none>
I20260924 12:47:00.731123 10642 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:00.731231 10642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:00.731527 10642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:00.734649 10642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "837d615d03414cbf9d7489b0ed6c0e4f"
format_stamp: "Formatted at 2026-09-24 12:47:00 on dist-test-slave-9tm1"
server_key: "cf1ea492329afb7063ae1e853fd05a28"
server_key_iv: "495dd86a924ce501e3a3693e9fa625b0"
server_key_version: "encryptionkey@0"
I20260924 12:47:00.735539 10642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "837d615d03414cbf9d7489b0ed6c0e4f"
format_stamp: "Formatted at 2026-09-24 12:47:00 on dist-test-slave-9tm1"
server_key: "cf1ea492329afb7063ae1e853fd05a28"
server_key_iv: "495dd86a924ce501e3a3693e9fa625b0"
server_key_version: "encryptionkey@0"
I20260924 12:47:00.741082 10642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 12:47:00.744740 10657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:00.746217 10642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 12:47:00.746394 10642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "837d615d03414cbf9d7489b0ed6c0e4f"
format_stamp: "Formatted at 2026-09-24 12:47:00 on dist-test-slave-9tm1"
server_key: "cf1ea492329afb7063ae1e853fd05a28"
server_key_iv: "495dd86a924ce501e3a3693e9fa625b0"
server_key_version: "encryptionkey@0"
I20260924 12:47:00.746590 10642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:00.767673 10642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:00.772338 10642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:00.772629 10642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:00.791667 10642 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45011
I20260924 12:47:00.791690 10708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45011 every 8 connection(s)
I20260924 12:47:00.793156 10642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:47:00.797631 10709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:00.803213 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 10642
I20260924 12:47:00.803431 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:47:00.804144 2373 external_mini_cluster.cc:1428] Setting key e5348eb818b0d15a498434af15fa7002
I20260924 12:47:00.809759 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Bootstrap starting.
I20260924 12:47:00.813599 10709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:00.814921 10709 log.cc:824] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:00.818876 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: No bootstrap required, opened a new log
Sep 24 12:47:00 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:00.828658 10709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:00.829208 10709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:00.829355 10709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837d615d03414cbf9d7489b0ed6c0e4f, State: Initialized, Role: FOLLOWER
I20260924 12:47:00.830219 10709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:00.830494 10709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:47:00.830648 10709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:47:00.830842 10709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:00.832044 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:00.806444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.2.81.126:45011)
0924 12:47:00.807253 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:00.807265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:00.807311 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:47:00.809097 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:00.809122 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:00.809569 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:00.809938 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:00.811774 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:00.813215 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:00.815049 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:00.816108 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:00.816408 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:00.822111 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:00.822124 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:00.822143 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:00.822200 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:00.826001 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:00.827004 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:00.827022 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:00.827031 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:00.827505 (+ 474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:00.828113 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:00.828120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:00.828122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:00.828774 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:00.829383 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:00.829842 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 12:47:00.830455 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":351,"threads_started":1}
I20260924 12:47:00.833487 10709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:00.834034 10709 leader_election.cc:302] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 837d615d03414cbf9d7489b0ed6c0e4f; no voters:
I20260924 12:47:00.834645 10709 leader_election.cc:288] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:47:00.834801 10714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:00.835903 10714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 LEADER]: Becoming Leader. State: Replica: 837d615d03414cbf9d7489b0ed6c0e4f, State: Running, Role: LEADER
I20260924 12:47:00.836418 10714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:00.836854 10709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:47:00.839357 10715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } } }
I20260924 12:47:00.839810 10715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:00.839362 10716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 837d615d03414cbf9d7489b0ed6c0e4f. Latest consensus state: current_term: 1 leader_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } } }
I20260924 12:47:00.840341 10716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:00.843842 10720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:47:00.849507 10720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:47:00.859844 10720 catalog_manager.cc:1347] Generated new cluster ID: 575cc7ade28044829b905c156e65c091
I20260924 12:47:00.860008 10720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:47:00.872227 10720 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:47:00.873703 10720 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:47:00.885242 10720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Generated new TSK 0
I20260924 12:47:00.886330 10720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:47:00.962008 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45011
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:01.315652 10735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:01.316058 10735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:01.316200 10735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:01.323719 10735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:01.323853 10735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:01.323994 10735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:47:01.334077 10735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:01.335366 10735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:01.336655 10735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:01.350252 10741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:01.350359 10743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:01.350277 10740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:01.351110 10735 server_base.cc:1034] running on GCE node
I20260924 12:47:01.351734 10735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:01.352797 10735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:01.354001 10735 hybrid_clock.cc:648] HybridClock initialized: now 1790254021353942 us; error 70 us; skew 500 ppm
I20260924 12:47:01.354313 10735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:01 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254021, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:01.361889 10735 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:47:01.363548 10735 webserver.cc:460] Webserver started at http://127.2.81.65:39999/ using document root <none> and password file <none>
I20260924 12:47:01.364248 10735 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:01.364370 10735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:01.364693 10735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:01.367641 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "c487976a48214aeda1b1b1acae5449bd"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "c0437b62b07d3a7b414e7c8124d72f65"
server_key_iv: "9a9cefe5b7599d6d1b21aef2505cd1d6"
server_key_version: "encryptionkey@0"
I20260924 12:47:01.368382 10735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "c487976a48214aeda1b1b1acae5449bd"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "c0437b62b07d3a7b414e7c8124d72f65"
server_key_iv: "9a9cefe5b7599d6d1b21aef2505cd1d6"
server_key_version: "encryptionkey@0"
I20260924 12:47:01.373682 10735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:47:01.377445 10750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:01.378906 10735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 12:47:01.379096 10735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "c487976a48214aeda1b1b1acae5449bd"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "c0437b62b07d3a7b414e7c8124d72f65"
server_key_iv: "9a9cefe5b7599d6d1b21aef2505cd1d6"
server_key_version: "encryptionkey@0"
I20260924 12:47:01.379293 10735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:01.411971 10735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:01.416304 10735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:01.416610 10735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:01.417654 10735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:01.419234 10735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:01.419327 10735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:01.419445 10735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:01.419505 10735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:01.451416 10735 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:33449
I20260924 12:47:01.451500 10862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:33449 every 8 connection(s)
I20260924 12:47:01.452742 10735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:47:01.455032 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 10735
I20260924 12:47:01.455199 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:47:01.455868 2373 external_mini_cluster.cc:1428] Setting key ea6951489a5710516b6456ab0efd054f
Sep 24 12:47:01 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254021, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:01.475698 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:01.455605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:41977 (local address 127.2.81.126:45011)
0924 12:47:01.456093 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:01.456099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:01.459347 (+ 3248us) server_negotiation.cc:407] Connection header received
0924 12:47:01.459404 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:01.459408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:01.459498 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:01.459579 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:01.461598 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.462297 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.464422 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.465083 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.465221 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:01.469583 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:01.469589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:01.469592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:01.469628 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:01.472223 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:01.473353 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:01.473359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:01.473362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:01.473500 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:01.473937 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:01.473941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:01.473943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:01.474211 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:01.474344 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:01.475096 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 12:47:01.475331 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":231,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:47:01.476716 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:01.456286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:41977)
0924 12:47:01.457234 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:01.457283 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:01.459039 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:01.459893 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:01.459906 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:01.460594 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:01.461412 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:01.461437 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.462454 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.462459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:01.464249 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:01.464265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.465349 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.465355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:01.465592 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:01.466465 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:01.466502 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:01.469307 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:01.472495 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:01.472505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:01.472522 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:01.473198 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:01.473616 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:01.473620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:01.473622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:01.473819 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:01.474335 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:01.474343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:01.474718 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 12:47:01.474969 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 12:47:01.475534 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":372,"threads_started":1}
I20260924 12:47:01.478487 10863 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45011
I20260924 12:47:01.478819 10863 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:47:01.479708 10863 heartbeater.cc:511] Master 127.2.81.126:45011 requested a full tablet report, sending...
I20260924 12:47:01.481997 10674 ts_manager.cc:194] Registered new tserver with Master: c487976a48214aeda1b1b1acae5449bd (127.2.81.65:33449)
I20260924 12:47:01.483515 10674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:41977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:47:01.508951 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45011
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:01.841316 10869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:01.841630 10869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:01.841737 10869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:01.851430 10869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:01.851581 10869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:01.851701 10869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:47:01.863010 10869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:01.864277 10869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:01.865620 10869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:01.879385 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:01.879290 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:01.879463 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:01.880749 10869 server_base.cc:1034] running on GCE node
I20260924 12:47:01.881451 10869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:01.882480 10869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:01.883947 10869 hybrid_clock.cc:648] HybridClock initialized: now 1790254021883842 us; error 82 us; skew 500 ppm
I20260924 12:47:01.884382 10869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:01 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254021, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:01.888779 10869 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:47:01.890825 10869 webserver.cc:460] Webserver started at http://127.2.81.66:36129/ using document root <none> and password file <none>
I20260924 12:47:01.891745 10869 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:01.891889 10869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:01.892267 10869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:01.896664 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "d8220296ff4a42b2a6dfc4ae8939c2e7"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "e068e6e57487d41bf94357688365fc17"
server_key_iv: "0ffdae5f59b9a66d0f705b1c1c50d327"
server_key_version: "encryptionkey@0"
I20260924 12:47:01.897740 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "d8220296ff4a42b2a6dfc4ae8939c2e7"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "e068e6e57487d41bf94357688365fc17"
server_key_iv: "0ffdae5f59b9a66d0f705b1c1c50d327"
server_key_version: "encryptionkey@0"
I20260924 12:47:01.905589 10869 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 12:47:01.911659 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:01.913606 10869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:47:01.913806 10869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "d8220296ff4a42b2a6dfc4ae8939c2e7"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "e068e6e57487d41bf94357688365fc17"
server_key_iv: "0ffdae5f59b9a66d0f705b1c1c50d327"
server_key_version: "encryptionkey@0"
I20260924 12:47:01.914049 10869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:01.937150 10869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:01.941610 10869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:01.941890 10869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:01.942850 10869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:01.944187 10869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:01.944262 10869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:01.944371 10869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:01.944417 10869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:01.976387 10869 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:34363
I20260924 12:47:01.976415 10996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:34363 every 8 connection(s)
I20260924 12:47:01.977893 10869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:47:01.981388 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 10869
I20260924 12:47:01.981680 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:47:01.982290 2373 external_mini_cluster.cc:1428] Setting key ca42cccf5eadfe31d3697d42a94fd63d
Sep 24 12:47:01 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254021, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:02.001926 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:01.980538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44795 (local address 127.2.81.126:45011)
0924 12:47:01.981122 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:01.981230 (+ 108us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:01.982536 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 12:47:01.985084 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:01.985089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:01.985167 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:01.985280 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:01.986957 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.987883 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.989633 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.990219 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.990356 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:01.994801 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:01.994807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:01.994810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:01.994849 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:01.997421 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:01.998757 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:01.998767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:01.998769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:01.998987 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:01.999649 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:01.999653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:01.999656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:01.999951 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:02.000095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:02.001358 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0924 12:47:02.001580 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
I20260924 12:47:02.003527 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:01.981250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:44795)
0924 12:47:01.982346 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:01.982401 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:01.984729 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:01.985559 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:01.985571 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:01.986172 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:01.986782 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:01.986802 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.988060 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.988064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:01.989462 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:01.989477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:01.990488 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:01.990495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:01.990754 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:01.991571 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:01.991602 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:01.994469 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:01.997596 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:01.997607 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:01.997627 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:01.998516 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:01.999186 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:01.999191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:01.999195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:01.999496 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:02.000106 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:02.000116 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:02.000644 (+ 528us) client_negotiation.cc:769] Sending connection context
0924 12:47:02.001038 (+ 394us) client_negotiation.cc:240] Negotiation successful
0924 12:47:02.001692 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":331,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:47:02.005425 10997 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45011
I20260924 12:47:02.005750 10997 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:02.006543 10997 heartbeater.cc:511] Master 127.2.81.126:45011 requested a full tablet report, sending...
I20260924 12:47:02.008085 10674 ts_manager.cc:194] Registered new tserver with Master: d8220296ff4a42b2a6dfc4ae8939c2e7 (127.2.81.66:34363)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:47:02.008960 10674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:44795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:47:02.036878 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45011
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:02.367879 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:02.368173 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:02.368265 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:02.375635 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:02.375767 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:02.375880 11003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:47:02.386198 11003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:02.387502 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:02.388741 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:02.401530 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:02.401609 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:02.401664 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:02.402635 11003 server_base.cc:1034] running on GCE node
I20260924 12:47:02.403311 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:02.404249 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:02.405478 11003 hybrid_clock.cc:648] HybridClock initialized: now 1790254022405387 us; error 53 us; skew 500 ppm
I20260924 12:47:02.405879 11003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:02 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254022, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:02.409112 11003 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:47:02.410651 11003 webserver.cc:460] Webserver started at http://127.2.81.67:33981/ using document root <none> and password file <none>
I20260924 12:47:02.411368 11003 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:02.411489 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:02.411788 11003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:02.414796 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "c4aaf80289984f2cb92b410adfae3e36"
format_stamp: "Formatted at 2026-09-24 12:47:02 on dist-test-slave-9tm1"
server_key: "6f35de05704b16e83a7b00079a725fdd"
server_key_iv: "32de9a138c1ebc84410fad4d30bae58c"
server_key_version: "encryptionkey@0"
I20260924 12:47:02.415583 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "c4aaf80289984f2cb92b410adfae3e36"
format_stamp: "Formatted at 2026-09-24 12:47:02 on dist-test-slave-9tm1"
server_key: "6f35de05704b16e83a7b00079a725fdd"
server_key_iv: "32de9a138c1ebc84410fad4d30bae58c"
server_key_version: "encryptionkey@0"
I20260924 12:47:02.421125 11003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:47:02.424661 11018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:02.426035 11003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:47:02.426193 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "c4aaf80289984f2cb92b410adfae3e36"
format_stamp: "Formatted at 2026-09-24 12:47:02 on dist-test-slave-9tm1"
server_key: "6f35de05704b16e83a7b00079a725fdd"
server_key_iv: "32de9a138c1ebc84410fad4d30bae58c"
server_key_version: "encryptionkey@0"
I20260924 12:47:02.426390 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:02.443823 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:02.448041 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:02.448303 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:02.449326 11003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:02.450700 11003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:02.450870 11003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:02.451014 11003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:02.451128 11003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:02.482614 11003 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:42677
I20260924 12:47:02.482674 11130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:42677 every 8 connection(s)
I20260924 12:47:02.484016 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:47:02.486637 10863 heartbeater.cc:503] Master 127.2.81.126:45011 was elected leader, sending a full tablet report...
I20260924 12:47:02.488583 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11003
I20260924 12:47:02.488729 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:47:02.489207 2373 external_mini_cluster.cc:1428] Setting key 451ff42f5a613cc210512a2db05875f7
Sep 24 12:47:02 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254022, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:02.503831 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:02.486416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:50085 (local address 127.2.81.126:45011)
0924 12:47:02.486626 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:02.486632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:02.487799 (+ 1167us) server_negotiation.cc:407] Connection header received
0924 12:47:02.489919 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:02.489924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:02.490010 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:02.490099 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:02.491632 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.492358 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.493941 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.494337 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.494443 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:02.497804 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:02.497811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:02.497814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:02.497851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:02.500329 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.501159 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.501165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.501169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.501289 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.501756 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.501762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.501765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.502124 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:02.502272 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:02.503402 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0924 12:47:02.503597 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 12:47:02.504817 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:02.486888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:50085)
0924 12:47:02.487649 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:02.487687 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:02.489646 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:02.490355 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:02.490364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:02.490875 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:02.491475 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:02.491500 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.492517 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.492521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:02.493805 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:02.493817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.494555 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.494563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:02.494775 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:02.495397 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:02.495424 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:02.497588 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:02.500496 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:02.500507 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:02.500525 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:02.500962 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:02.501394 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:02.501398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:02.501401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:02.501641 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:02.502450 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:02.502460 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:02.502890 (+ 430us) client_negotiation.cc:769] Sending connection context
0924 12:47:02.503166 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 12:47:02.503723 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":312,"threads_started":1}
I20260924 12:47:02.506153 11131 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45011
I20260924 12:47:02.506484 11131 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:02.507218 11131 heartbeater.cc:511] Master 127.2.81.126:45011 requested a full tablet report, sending...
I20260924 12:47:02.508616 10674 ts_manager.cc:194] Registered new tserver with Master: c4aaf80289984f2cb92b410adfae3e36 (127.2.81.67:42677)
I20260924 12:47:02.509482 10674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:50085
I20260924 12:47:02.516487 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab.
Entry for principal oryx/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab.
WARNING: no policy specified for oryx/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab.
Entry for principal oryx/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab.
WARNING: no policy specified for oryx/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab.
Entry for principal oryx/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab.
I20260924 12:47:02.656540 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 10735
Sep 24 12:47:02 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:47:02.672878 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:02.657635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.2.81.65:33449)
0924 12:47:02.658207 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:02.658215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:02.658243 (+ 28us) server_negotiation.cc:407] Connection header received
0924 12:47:02.658351 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:02.658358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:02.658567 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:02.658718 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:02.660024 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.661117 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.662422 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.663007 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.663107 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:02.667215 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:02.667229 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:02.667245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:02.667288 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:02.670086 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.670810 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.670818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.670820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.670962 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.671331 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.671334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.671336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.671686 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:02.671997 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:02.672271 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 12:47:02.672529 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":274,"threads_started":1}
I20260924 12:47:02.674073 10858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:02.755867 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 10735
I20260924 12:47:02.768529 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 10869
Sep 24 12:47:02 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:47:02.785285 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:02.769412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.2.81.66:34363)
0924 12:47:02.769868 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:02.769875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:02.769899 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:47:02.769957 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:02.769963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:02.770188 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:02.770318 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:02.771518 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.772532 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.773935 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.774518 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.774628 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:02.778827 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:02.778842 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:02.778860 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:02.778900 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:02.781825 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.782680 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.782688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.782690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.782908 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.783356 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.783361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.783363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.783741 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:02.784040 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:02.784473 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 12:47:02.784790 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260924 12:47:02.786567 10992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:02.870844 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 10869
I20260924 12:47:02.883142 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11003
Sep 24 12:47:02 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:47:02.899394 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:02.884093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.2.81.67:42677)
0924 12:47:02.884756 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:02.884763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:02.884792 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:47:02.884852 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:02.884858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:02.885128 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:02.885286 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:02.886561 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.887601 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.888849 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:02.889466 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:02.889581 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:02.893830 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:02.893844 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:02.893861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:02.893899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:02.896594 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.897359 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.897367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.897370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.897515 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:02.897860 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:02.897864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:02.897866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:02.898217 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:02.898520 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:02.898789 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 12:47:02.899041 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":371,"threads_started":1}
I20260924 12:47:02.900676 11126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:02.985383 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11003
I20260924 12:47:02.997756 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 10642
I20260924 12:47:02.999053 10704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:03.098709 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 10642
I20260924 12:47:03.115473 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45011
--webserver_interface=127.2.81.126
--webserver_port=39493
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:03.463099 11148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:03.463840 11148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:03.464036 11148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:03.473450 11148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:47:03.473635 11148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:03.473711 11148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:03.473793 11148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:47:03.473866 11148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:47:03.486204 11148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=39493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:03.487634 11148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:03.489301 11148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:03.502929 11154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:03.502729 11156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:03.502951 11153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:03.503875 11148 server_base.cc:1034] running on GCE node
I20260924 12:47:03.504709 11148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:03.506475 11148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:03.507732 11148 hybrid_clock.cc:648] HybridClock initialized: now 1790254023507652 us; error 74 us; skew 500 ppm
I20260924 12:47:03.508108 11148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:03 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254023, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:03.511554 11148 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:47:03.513339 11148 webserver.cc:460] Webserver started at http://127.2.81.126:39493/ using document root <none> and password file <none>
I20260924 12:47:03.514077 11148 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:03.514211 11148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:03.520125 11148 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 12:47:03.523171 11163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:03.524603 11148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 12:47:03.524776 11148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "837d615d03414cbf9d7489b0ed6c0e4f"
format_stamp: "Formatted at 2026-09-24 12:47:00 on dist-test-slave-9tm1"
server_key: "cf1ea492329afb7063ae1e853fd05a28"
server_key_iv: "495dd86a924ce501e3a3693e9fa625b0"
server_key_version: "encryptionkey@0"
I20260924 12:47:03.525873 11148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:03.553890 11148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:03.558490 11148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:03.558760 11148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:03.577522 11148 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45011
I20260924 12:47:03.577561 11214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45011 every 8 connection(s)
I20260924 12:47:03.579257 11148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:47:03.586606 11215 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:47:03.588140 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11148
I20260924 12:47:03.588959 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:33449
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:45011
--webserver_port=39999
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:47:03.590751 11215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Bootstrap starting.
I20260924 12:47:03.619161 11215 log.cc:824] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:03.629837 11215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:47:03.630705 11215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Bootstrap complete.
I20260924 12:47:03.640435 11215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:03.641376 11215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837d615d03414cbf9d7489b0ed6c0e4f, State: Initialized, Role: FOLLOWER
I20260924 12:47:03.641987 11215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:03.642217 11215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:47:03.642326 11215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:47:03.642488 11215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:47:03.646899 11215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:03.647554 11215 leader_election.cc:302] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 837d615d03414cbf9d7489b0ed6c0e4f; no voters:
I20260924 12:47:03.648339 11215 leader_election.cc:288] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 12:47:03.648483 11219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:47:03.650604 11215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:47:03.656004 11219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [term 2 LEADER]: Becoming Leader. State: Replica: 837d615d03414cbf9d7489b0ed6c0e4f, State: Running, Role: LEADER
I20260924 12:47:03.657622 11219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } }
I20260924 12:47:03.661751 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 837d615d03414cbf9d7489b0ed6c0e4f. Latest consensus state: current_term: 2 leader_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } } }
I20260924 12:47:03.662243 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:03.662777 11220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837d615d03414cbf9d7489b0ed6c0e4f" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45011 } } }
I20260924 12:47:03.663255 11220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:03.670720 11230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:47:03.677754 11230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:47:03.680030 11230 catalog_manager.cc:1259] Loaded cluster ID: 575cc7ade28044829b905c156e65c091
I20260924 12:47:03.680291 11230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:47:03.684911 11230 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:47:03.686828 11230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 837d615d03414cbf9d7489b0ed6c0e4f: Loaded TSK: 0
I20260924 12:47:03.689524 11230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 12:47:03.962280 11217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:03.962594 11217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:03.962715 11217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:03.970917 11217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:03.971076 11217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:03.971194 11217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:47:03.984048 11217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.65.keytab
--principal=oryx/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:33449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=39999
--tserver_master_addrs=127.2.81.126:45011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:03.985491 11217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:03.986842 11217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:04.003998 11245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:04.004257 11242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:04.004034 11243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:04.005810 11217 server_base.cc:1034] running on GCE node
I20260924 12:47:04.006649 11217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:04.008044 11217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:04.009418 11217 hybrid_clock.cc:648] HybridClock initialized: now 1790254024009360 us; error 73 us; skew 500 ppm
I20260924 12:47:04.009822 11217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254024, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:04.013924 11217 init.cc:377] Logged in from keytab as oryx/127.2.81.65@KRBTEST.COM (short username oryx)
I20260924 12:47:04.015753 11217 webserver.cc:460] Webserver started at http://127.2.81.65:39999/ using document root <none> and password file <none>
I20260924 12:47:04.016533 11217 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:04.016680 11217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:04.023422 11217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:47:04.027477 11252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:04.029182 11217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:47:04.029382 11217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "c487976a48214aeda1b1b1acae5449bd"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "c0437b62b07d3a7b414e7c8124d72f65"
server_key_iv: "9a9cefe5b7599d6d1b21aef2505cd1d6"
server_key_version: "encryptionkey@0"
I20260924 12:47:04.030478 11217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:04.063803 11217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:04.069582 11217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:04.069871 11217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:04.070958 11217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:04.072468 11217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:04.072580 11217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:04.072661 11217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:04.072736 11217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:04.109023 11217 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:33449
I20260924 12:47:04.109112 11364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:33449 every 8 connection(s)
I20260924 12:47:04.110592 11217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:47:04.116680 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11217
I20260924 12:47:04.117381 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:34363
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:45011
--webserver_port=36129
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 12:47:04.136214 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:04.114270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:51675 (local address 127.2.81.126:45011)
0924 12:47:04.115223 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:04.115241 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:04.115552 (+ 311us) server_negotiation.cc:407] Connection header received
0924 12:47:04.117993 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:04.118015 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:04.118578 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:04.119055 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:04.121112 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.123766 (+ 2654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.127014 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.128099 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.128345 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.134892 (+ 6547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:51675: BlockingRecv error: recv got EOF from 127.2.81.65:51675 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":378,"threads_started":1}
I20260924 12:47:04.136190 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:04.114408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:51675)
0924 12:47:04.115277 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:04.115324 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:04.117250 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:04.119250 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:04.119259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:04.119794 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:04.120621 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.120644 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.124383 (+ 3739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.124390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.126724 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.126755 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.128533 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.128544 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.128849 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.129687 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:04.129717 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:04.134357 (+ 4640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":333,"threads_started":1}
W20260924 12:47:04.137207 11365 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45011 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.81.126:45011: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:04.147348 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:04.137893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:41017)
0924 12:47:04.138079 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:04.138096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:04.138217 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:04.138560 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:04.138566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:04.138917 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:04.139068 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.139078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.140170 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.140174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.141668 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.141687 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.142455 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.142461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.142640 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.143414 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:04.143434 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:04.147051 (+ 3617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 12:47:04.147799 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:04.138074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:41017 (local address 127.2.81.126:45011)
0924 12:47:04.138258 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:04.138264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:04.138298 (+ 34us) server_negotiation.cc:407] Connection header received
0924 12:47:04.138358 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:04.138362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:04.138419 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:04.138501 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:04.139226 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.140000 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.141870 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.142292 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.142418 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.147310 (+ 4892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:41017: BlockingRecv error: recv got EOF from 127.2.81.65:41017 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:04.156728 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:04.148159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:43153)
0924 12:47:04.148330 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:04.148351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:04.148472 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:04.148816 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:04.148821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:04.149200 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:04.149366 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.149375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.150418 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.150423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.151465 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.151479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.152319 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.152325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.152531 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.153197 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:04.153216 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:04.156408 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 12:47:04.157111 11365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 12:47:04.157220 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:04.148237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:43153 (local address 127.2.81.126:45011)
0924 12:47:04.148435 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:04.148440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:04.148459 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:47:04.148606 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:04.148610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:04.148686 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:04.148774 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:04.149521 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.150268 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.151652 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.152161 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.152274 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.156677 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:43153: BlockingRecv error: recv got EOF from 127.2.81.65:43153 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 12:47:04.483423 11369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:04.483749 11369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:04.483873 11369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:04.491622 11369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:04.491794 11369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:04.491964 11369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:47:04.503043 11369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.66.keytab
--principal=oryx/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:34363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=36129
--tserver_master_addrs=127.2.81.126:45011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:04.504830 11369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:04.506330 11369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:04.520447 11374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:04.520398 11375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:04.520483 11377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:04.521101 11369 server_base.cc:1034] running on GCE node
I20260924 12:47:04.521750 11369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:04.522751 11369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:04.523978 11369 hybrid_clock.cc:648] HybridClock initialized: now 1790254024523905 us; error 57 us; skew 500 ppm
I20260924 12:47:04.524323 11369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254024, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:04.527602 11369 init.cc:377] Logged in from keytab as oryx/127.2.81.66@KRBTEST.COM (short username oryx)
I20260924 12:47:04.529175 11369 webserver.cc:460] Webserver started at http://127.2.81.66:36129/ using document root <none> and password file <none>
I20260924 12:47:04.529870 11369 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:04.529997 11369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:04.535912 11369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:47:04.539150 11384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:04.540846 11369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:47:04.541085 11369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "d8220296ff4a42b2a6dfc4ae8939c2e7"
format_stamp: "Formatted at 2026-09-24 12:47:01 on dist-test-slave-9tm1"
server_key: "e068e6e57487d41bf94357688365fc17"
server_key_iv: "0ffdae5f59b9a66d0f705b1c1c50d327"
server_key_version: "encryptionkey@0"
I20260924 12:47:04.542222 11369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:04.567237 11369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:04.571853 11369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:04.572139 11369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:04.573287 11369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:04.574659 11369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:04.574769 11369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:04.574873 11369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:04.574941 11369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:04.615403 11369 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:34363
I20260924 12:47:04.615486 11496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:34363 every 8 connection(s)
I20260924 12:47:04.616801 11369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:47:04.617875 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11369
I20260924 12:47:04.618582 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:42677
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:45011
--webserver_port=33981
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 12:47:04.636494 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:04.619603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:39475 (local address 127.2.81.126:45011)
0924 12:47:04.619801 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:04.619807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:04.621144 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 12:47:04.623078 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:04.623084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:04.623173 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:04.623288 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:04.625851 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.626996 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.628860 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.629663 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.629807 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.636290 (+ 6483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:39475: BlockingRecv error: recv got EOF from 127.2.81.66:39475 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:47:04.637537 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:04.620108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:39475)
0924 12:47:04.620928 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:04.621008 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:04.622764 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:04.623583 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:04.623593 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:04.624410 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:04.625600 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.625629 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.627247 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.627252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.628615 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.628631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.630007 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.630015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.630443 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.631193 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:04.631229 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:04.636012 (+ 4783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":354,"threads_started":1}
W20260924 12:47:04.638477 11497 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45011 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.81.126:45011: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:04.648357 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:04.639205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:55967)
0924 12:47:04.639516 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:04.639533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:04.639630 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:04.640015 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:04.640019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:04.640349 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:04.640494 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.640502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.642125 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.642130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.643108 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.643124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.644167 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.644171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.644303 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.644922 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:04.644941 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:04.648121 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190}
W20260924 12:47:04.648725 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:04.639367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:55967 (local address 127.2.81.126:45011)
0924 12:47:04.639550 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:04.639556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:04.639579 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:47:04.639746 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:04.639752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:04.639833 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:04.639953 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:04.640695 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.641915 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.643321 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.643989 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.644124 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.648480 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:55967: BlockingRecv error: recv got EOF from 127.2.81.66:55967 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:04 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:04.657927 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:04.649067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:46879)
0924 12:47:04.649412 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:04.649428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:04.649537 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:04.649831 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:04.649834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:04.650135 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:04.650319 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.650331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.651833 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.651838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.652873 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:04.652890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.653820 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.653825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:04.653977 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.654641 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:04.654660 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:04.657699 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235}
W20260924 12:47:04.658212 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:04.649268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:46879 (local address 127.2.81.126:45011)
0924 12:47:04.649432 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:04.649439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:04.649486 (+ 47us) server_negotiation.cc:407] Connection header received
0924 12:47:04.649635 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:04.649640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:04.649714 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:04.649816 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:04.650460 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.651636 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.653076 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:04.653652 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:04.653784 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:04.657980 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:46879: BlockingRecv error: recv got EOF from 127.2.81.66:46879 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 12:47:04.658368 11497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 12:47:04.965206 11500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:04.965495 11500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:04.965590 11500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:04.973085 11500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:04.973208 11500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:04.973325 11500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:47:04.983794 11500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/krb5kdc/oryx_127.2.81.67.keytab
--principal=oryx/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:42677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=33981
--tserver_master_addrs=127.2.81.126:45011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:04.985177 11500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:04.986699 11500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:05.000905 11505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:05.000747 11506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:05.001824 11508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:05.003783 11500 server_base.cc:1034] running on GCE node
I20260924 12:47:05.004513 11500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:05.005882 11500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:05.007248 11500 hybrid_clock.cc:648] HybridClock initialized: now 1790254025007171 us; error 69 us; skew 500 ppm
I20260924 12:47:05.007612 11500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254025, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:05.011119 11500 init.cc:377] Logged in from keytab as oryx/127.2.81.67@KRBTEST.COM (short username oryx)
I20260924 12:47:05.012790 11500 webserver.cc:460] Webserver started at http://127.2.81.67:33981/ using document root <none> and password file <none>
I20260924 12:47:05.013592 11500 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:05.013734 11500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:05.019820 11500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:47:05.023531 11515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:05.025966 11500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:47:05.026382 11500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "c4aaf80289984f2cb92b410adfae3e36"
format_stamp: "Formatted at 2026-09-24 12:47:02 on dist-test-slave-9tm1"
server_key: "6f35de05704b16e83a7b00079a725fdd"
server_key_iv: "32de9a138c1ebc84410fad4d30bae58c"
server_key_version: "encryptionkey@0"
I20260924 12:47:05.027559 11500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:05.055984 11500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:05.061899 11500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:05.062311 11500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:05.063654 11500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:05.065197 11500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:05.065299 11500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:05.065410 11500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:05.065476 11500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:05.099485 11500 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:42677
I20260924 12:47:05.099506 11627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:42677 every 8 connection(s)
I20260924 12:47:05.100926 11500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:47:05.104025 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11500
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 12:47:05.118886 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:05.103525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:39977 (local address 127.2.81.126:45011)
0924 12:47:05.103785 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:05.103792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:05.104946 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 12:47:05.107182 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:05.107187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:05.107258 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:05.107338 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:05.109488 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.110610 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.112648 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.113105 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.113211 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.118630 (+ 5419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:39977: BlockingRecv error: recv got EOF from 127.2.81.67:39977 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 12:47:05.119582 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:05.103954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:39977)
0924 12:47:05.104754 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:05.104806 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:05.106906 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:05.107687 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:05.107698 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:05.108409 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:05.109264 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.109293 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.110783 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.110789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.112439 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.112453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.113321 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.113329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.113548 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.114132 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:05.114155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:05.118405 (+ 4250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":324,"threads_started":1}
W20260924 12:47:05.120386 11628 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45011 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.81.126:45011: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
I20260924 12:47:05.122452 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:05.104911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.2.81.126:45011)
0924 12:47:05.105528 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:05.105533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:05.105900 (+ 367us) server_negotiation.cc:407] Connection header received
0924 12:47:05.106033 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:05.106039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:05.106115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:05.106231 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:05.108046 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.109322 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.110631 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.111234 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.111360 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.114779 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:05.114790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:05.114811 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:05.114855 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:05.118555 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:05.119410 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:05.119420 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:05.119428 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:05.119648 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:05.120128 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:05.120132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:05.120135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:05.120720 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:05.121072 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:05.121650 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 12:47:05.122063 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":508,"thread_start_us":392,"threads_started":1}
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:05.129832 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:05.120997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:39671)
0924 12:47:05.121288 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:05.121306 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:05.121444 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:05.121784 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:05.121789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:05.122177 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:05.122357 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.122368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.124408 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.124413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.125512 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.125527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.126185 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.126191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.126351 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.126893 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:05.126912 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:05.129604 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107}
W20260924 12:47:05.130442 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:05.121114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:39671 (local address 127.2.81.126:45011)
0924 12:47:05.121286 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:05.121291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:05.121359 (+ 68us) server_negotiation.cc:407] Connection header received
0924 12:47:05.121565 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:05.121570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:05.121639 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:05.121737 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:05.122546 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.123407 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.125652 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.126049 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.126155 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.129857 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:39671: BlockingRecv error: recv got EOF from 127.2.81.67:39671 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":71}
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:05.139426 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:05.130548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:34015)
0924 12:47:05.130710 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:05.130729 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:05.130825 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:05.131200 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:05.131204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:05.131472 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:05.131626 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.131635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.132751 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.132755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.133785 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.133799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.134486 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.134490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.134621 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.135261 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:05.135280 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:05.138806 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 12:47:05.139509 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:05.130651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:34015 (local address 127.2.81.126:45011)
0924 12:47:05.130848 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:05.130854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:05.130874 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:47:05.130986 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:05.130989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:05.131057 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:05.131138 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:05.131786 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.132604 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.133956 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.134342 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.134449 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.139276 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:34015: BlockingRecv error: recv got EOF from 127.2.81.67:34015 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260924 12:47:05.139786 11628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:05.167212 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:05.157869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:54875)
0924 12:47:05.158516 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:05.158536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:05.158719 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:05.159063 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:05.159067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:05.159574 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:05.159806 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.159817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.160893 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.160898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.162006 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.162024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.162883 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.162890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.163121 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.163839 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:05.163863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:05.166977 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":590,"threads_started":1}
W20260924 12:47:05.167465 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:05.158015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:54875 (local address 127.2.81.126:45011)
0924 12:47:05.158233 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:05.158240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:05.158641 (+ 401us) server_negotiation.cc:407] Connection header received
0924 12:47:05.158832 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:05.158836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:05.158899 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:05.158988 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:05.159949 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.160734 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.162213 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.162650 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.162786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.167201 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:54875: BlockingRecv error: recv got EOF from 127.2.81.65:54875 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:05 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:05.668440 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:05.659153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:39377)
0924 12:47:05.659640 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:05.659660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:05.659780 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:05.660144 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:05.660148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:05.660592 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:05.660780 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.660789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.661911 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.661916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.663101 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:05.663117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.664166 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.664172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:05.664334 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.665035 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:05.665054 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:05.668123 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":241,"threads_started":1}
W20260924 12:47:05.668613 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:05.659280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:39377 (local address 127.2.81.126:45011)
0924 12:47:05.659443 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:05.659447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:05.659731 (+ 284us) server_negotiation.cc:407] Connection header received
0924 12:47:05.659881 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:05.659885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:05.659992 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:05.660094 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:05.660946 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.661723 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.663257 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:05.663968 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:05.664100 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:05.668371 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:39377: BlockingRecv error: recv got EOF from 127.2.81.66:39377 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 12:47:06 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:06 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:06.150367 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:06.140561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:35269)
0924 12:47:06.141213 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:06.141229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:06.141355 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:06.141655 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:06.141658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:06.142131 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:06.142324 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:06.142334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.143525 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.143530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:06.144807 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:06.144827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.145913 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.145924 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:06.146105 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:06.146813 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:06.146833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:06.150129 (+ 3296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":394,"threads_started":1}
W20260924 12:47:06.150718 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:06.140704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:35269 (local address 127.2.81.126:45011)
0924 12:47:06.140923 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:06.140927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:06.141326 (+ 399us) server_negotiation.cc:407] Connection header received
0924 12:47:06.141456 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:06.141460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:06.141527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:06.141621 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:06.142509 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.143313 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.145114 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.145710 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.145852 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:06.150444 (+ 4592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:35269: BlockingRecv error: recv got EOF from 127.2.81.67:35269 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 12:47:06 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:06 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:06.178885 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:06.168288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:60747)
0924 12:47:06.168917 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:06.168940 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:06.169194 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:06.169736 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:06.169741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:06.170461 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:06.170658 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:06.170669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.172178 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.172183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:06.173280 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:06.173299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.174144 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.174150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:06.174381 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:06.175184 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:06.175204 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:06.178627 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":275,"threads_started":1}
W20260924 12:47:06.179431 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:06.168451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:60747 (local address 127.2.81.126:45011)
0924 12:47:06.168714 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:06.168721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:06.169069 (+ 348us) server_negotiation.cc:407] Connection header received
0924 12:47:06.169414 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:06.169419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:06.169520 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:06.169616 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:06.170837 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.171965 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.173467 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.173895 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.174006 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:06.179121 (+ 5115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:60747: BlockingRecv error: recv got EOF from 127.2.81.65:60747 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 24 12:47:06 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:06 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:06.681546 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:06.669476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:60553)
0924 12:47:06.669983 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:06.670003 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:06.670156 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:06.670463 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:06.670468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:06.670876 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:06.671014 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:06.671023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.672146 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.672150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:06.673114 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:06.673126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.673940 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.673946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:06.674103 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:06.678342 (+ 4239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:06.678368 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:06.681291 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":285,"threads_started":1}
W20260924 12:47:06.681756 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:06.669609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:60553 (local address 127.2.81.126:45011)
0924 12:47:06.669806 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:06.669812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:06.670099 (+ 287us) server_negotiation.cc:407] Connection header received
0924 12:47:06.670274 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:06.670277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:06.670336 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:06.670415 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:06.671217 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.671961 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.673322 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:06.673781 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:06.673900 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:06.681497 (+ 7597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:60553: BlockingRecv error: recv got EOF from 127.2.81.66:60553 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 12:47:07 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:07 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:07.160699 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:07.151586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:54031)
0924 12:47:07.152201 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:07.152221 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:07.152339 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:07.152745 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:07.152749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:07.153180 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:07.153327 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:07.153336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.154575 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.154579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:07.155536 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:07.155551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.156517 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.156522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:07.156669 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:07.157299 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:07.157319 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:07.160446 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":357,"threads_started":1}
W20260924 12:47:07.161031 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:07.151646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:54031 (local address 127.2.81.126:45011)
0924 12:47:07.151877 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:07.151885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:07.152307 (+ 422us) server_negotiation.cc:407] Connection header received
0924 12:47:07.152476 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:07.152480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:07.152546 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:07.152635 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:07.153485 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.154298 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.155725 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.156323 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.156444 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:07.160741 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:54031: BlockingRecv error: recv got EOF from 127.2.81.67:54031 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 12:47:07 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:07 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:07.188612 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:07.179869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:39873)
0924 12:47:07.180392 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:07.180408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:07.180570 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:07.180903 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:07.180908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:07.181289 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:07.181602 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:07.181617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.182616 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.182622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:07.183540 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:07.183556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.184235 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.184242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:07.184407 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:07.185084 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:07.185104 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:07.188268 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":305,"threads_started":1}
W20260924 12:47:07.189205 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:07.180029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:39873 (local address 127.2.81.126:45011)
0924 12:47:07.180252 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:07.180259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:07.180492 (+ 233us) server_negotiation.cc:407] Connection header received
0924 12:47:07.180694 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:07.180699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:07.180776 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:07.180858 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:07.181776 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.182474 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.183693 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.184091 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.184202 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:07.188564 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:39873: BlockingRecv error: recv got EOF from 127.2.81.65:39873 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 12:47:07 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:07 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:07.690831 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:07.682569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:46197)
0924 12:47:07.683168 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:07.683190 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:07.683328 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:07.683651 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:07.683656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:07.684017 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:07.684185 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:07.684194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.685275 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.685281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:07.686198 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:07.686212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.686891 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.686898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:07.687076 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:07.687678 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:07.687696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:07.690571 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":399,"threads_started":1}
W20260924 12:47:07.691115 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:07.682702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:46197 (local address 127.2.81.126:45011)
0924 12:47:07.682997 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:07.683003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:07.683274 (+ 271us) server_negotiation.cc:407] Connection header received
0924 12:47:07.683439 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:07.683444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:07.683512 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:07.683593 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:07.684333 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.685142 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.686353 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:07.686742 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:07.686852 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:07.690845 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:46197: BlockingRecv error: recv got EOF from 127.2.81.66:46197 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 12:47:08 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:08 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:08.169831 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:08.161601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:42533)
0924 12:47:08.162130 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:08.162151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:08.162303 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:08.162668 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:08.162672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:08.163090 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:08.163256 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:08.163267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.164271 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.164276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:08.165213 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:08.165225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.165930 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.165935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:08.166105 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:08.166699 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:08.166717 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:08.169553 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
W20260924 12:47:08.169972 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:08.161759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:42533 (local address 127.2.81.126:45011)
0924 12:47:08.161983 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:08.161989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:08.162258 (+ 269us) server_negotiation.cc:407] Connection header received
0924 12:47:08.162464 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:08.162468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:08.162532 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:08.162615 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:08.163413 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.164125 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.165356 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.165778 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.165891 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:08.169796 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:42533: BlockingRecv error: recv got EOF from 127.2.81.67:42533 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 12:47:08 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:08 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:08.198618 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:08.189653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:43959)
0924 12:47:08.190195 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:08.190210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:08.190363 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:08.190729 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:08.190735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:08.191138 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:08.191313 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:08.191323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.192350 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.192354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:08.193413 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:08.193428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.194183 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.194189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:08.194369 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:08.195099 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:08.195117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:08.198358 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":336,"threads_started":1}
W20260924 12:47:08.198825 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:08.189770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:43959 (local address 127.2.81.126:45011)
0924 12:47:08.190013 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:08.190019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:08.190294 (+ 275us) server_negotiation.cc:407] Connection header received
0924 12:47:08.190496 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:08.190502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:08.190579 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:08.190667 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:08.191460 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.192206 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.193581 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.194022 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.194147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:08.198616 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:43959: BlockingRecv error: recv got EOF from 127.2.81.65:43959 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 12:47:08 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:08 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:08.700314 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:08.691780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:53303)
0924 12:47:08.692327 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:08.692347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:08.692512 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:08.692820 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:08.692824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:08.693245 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:08.693388 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:08.693398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.694389 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.694394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:08.695455 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:08.695470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.696223 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.696228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:08.696400 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:08.697090 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:08.697115 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:08.699942 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":310,"threads_started":1}
W20260924 12:47:08.700381 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:08.691927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:53303 (local address 127.2.81.126:45011)
0924 12:47:08.692221 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:08.692227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:08.692431 (+ 204us) server_negotiation.cc:407] Connection header received
0924 12:47:08.692633 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:08.692637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:08.692693 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:08.692773 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:08.693539 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.694256 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.695601 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:08.696089 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:08.696196 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:08.700169 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:53303: BlockingRecv error: recv got EOF from 127.2.81.66:53303 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 24 12:47:09 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:09 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:09.180059 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:09.171017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:35317)
0924 12:47:09.171566 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:09.171585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:09.171732 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:09.172146 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:09.172149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:09.172562 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:09.172713 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:09.172722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.173902 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.173906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:09.174920 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:09.174934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.175703 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.175708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:09.175851 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:09.176662 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:09.176681 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:09.179804 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":267,"threads_started":1}
W20260924 12:47:09.180289 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:09.171121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:35317 (local address 127.2.81.126:45011)
0924 12:47:09.171310 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:09.171317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:09.171659 (+ 342us) server_negotiation.cc:407] Connection header received
0924 12:47:09.171897 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:09.171903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:09.171975 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:09.172062 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:09.172876 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.173744 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.175086 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.175514 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.175657 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:09.180059 (+ 4402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:35317: BlockingRecv error: recv got EOF from 127.2.81.67:35317 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 12:47:09 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:09 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:09.209329 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:09.199669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:57463)
0924 12:47:09.200146 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:09.200162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:09.200288 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:09.200747 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:09.200751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:09.201370 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:09.201558 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:09.201570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.202809 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.202813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:09.203807 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:09.203823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.204670 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.204675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:09.204820 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:09.205740 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:09.205759 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:09.209005 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":224,"threads_started":1}
W20260924 12:47:09.209512 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:09.199763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:57463 (local address 127.2.81.126:45011)
0924 12:47:09.199968 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:09.199979 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:09.200269 (+ 290us) server_negotiation.cc:407] Connection header received
0924 12:47:09.200477 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:09.200482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:09.200567 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:09.200682 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:09.201790 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.202633 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.204010 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.204495 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.204605 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:09.209310 (+ 4705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:57463: BlockingRecv error: recv got EOF from 127.2.81.65:57463 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 12:47:09 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:09 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:09.710565 11646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:09.701327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:35107)
0924 12:47:09.701867 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:09.701890 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:09.702031 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:09.702338 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:09.702343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:09.702824 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:09.703013 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:09.703023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.704057 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.704063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:09.705031 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:09.705046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.705776 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.705782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:09.705960 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:09.706657 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:09.706679 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:09.710111 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":266,"threads_started":1}
W20260924 12:47:09.710723 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:09.701589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:35107 (local address 127.2.81.126:45011)
0924 12:47:09.701819 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:09.701825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:09.701980 (+ 155us) server_negotiation.cc:407] Connection header received
0924 12:47:09.702144 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:09.702149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:09.702208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:09.702289 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:09.703162 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.703905 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.705188 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:09.705620 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:09.705739 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:09.710407 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:35107: BlockingRecv error: recv got EOF from 127.2.81.66:35107 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 12:47:10 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:10 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:10.191256 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:10.181120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:57087)
0924 12:47:10.181638 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:10.181654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:10.181770 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:10.182210 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:10.182214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:10.182624 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:10.182891 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:10.182906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.184248 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.184253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:10.185453 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:10.185477 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.186331 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.186335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:10.186478 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:10.187209 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:10.187235 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:10.190888 (+ 3653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":243,"threads_started":1}
W20260924 12:47:10.191356 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:10.181171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:57087 (local address 127.2.81.126:45011)
0924 12:47:10.181349 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:10.181354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:10.181737 (+ 383us) server_negotiation.cc:407] Connection header received
0924 12:47:10.181910 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:10.181915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:10.181988 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:10.182155 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:10.183202 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.184073 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.185669 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.186143 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.186251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:10.191143 (+ 4892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:57087: BlockingRecv error: recv got EOF from 127.2.81.67:57087 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 12:47:10 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:10 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:10.218503 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:10.210394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:54505)
0924 12:47:10.210840 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:10.210856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:10.210992 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:10.211325 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:10.211330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:10.211712 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:10.211858 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:10.211882 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.212916 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.212920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:10.214023 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:10.214036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.214753 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.214759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:10.214958 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:10.215613 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:10.215630 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:10.218282 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":254,"threads_started":1}
W20260924 12:47:10.218895 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:10.210505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:54505 (local address 127.2.81.126:45011)
0924 12:47:10.210731 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:10.210738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:10.210942 (+ 204us) server_negotiation.cc:407] Connection header received
0924 12:47:10.211115 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:10.211119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:10.211177 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:10.211263 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:10.212006 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.212759 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.214136 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.214610 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.214740 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:10.218515 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:54505: BlockingRecv error: recv got EOF from 127.2.81.65:54505 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 12:47:10 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:10 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:10.720315 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:10.711789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:59907)
0924 12:47:10.712327 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:10.712347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:10.712469 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:10.712878 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:10.712881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:10.713256 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:10.713421 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:10.713437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.714476 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.714480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:10.715405 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:10.715417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.716371 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.716378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:10.716549 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:10.717216 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:10.717234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:10.720083 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
W20260924 12:47:10.720628 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:10.711921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:59907 (local address 127.2.81.126:45011)
0924 12:47:10.712160 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:10.712165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:10.712434 (+ 269us) server_negotiation.cc:407] Connection header received
0924 12:47:10.712641 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:10.712647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:10.712727 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:10.712846 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:10.713590 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.714318 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.715526 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:10.716163 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:10.716294 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:10.720384 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:59907: BlockingRecv error: recv got EOF from 127.2.81.66:59907 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 12:47:11 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:11 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:11.200893 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:11.192291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:48677)
0924 12:47:11.192804 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:11.192819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:11.192939 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:11.193309 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:11.193313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:11.193664 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:11.193840 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:11.193849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.194977 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.194982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:11.195949 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:11.195965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.196780 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.196785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:11.196937 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:11.197659 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:11.197679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:11.200670 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":297,"threads_started":1}
W20260924 12:47:11.201292 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:11.192362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:48677 (local address 127.2.81.126:45011)
0924 12:47:11.192530 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:11.192535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:11.192899 (+ 364us) server_negotiation.cc:407] Connection header received
0924 12:47:11.193080 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:11.193084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:11.193147 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:11.193235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:11.194029 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.194807 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.196125 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.196585 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.196696 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:11.200880 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:48677: BlockingRecv error: recv got EOF from 127.2.81.67:48677 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 12:47:11 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:11 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:11.228318 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:11.219414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:38137)
0924 12:47:11.220004 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:11.220022 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:11.220193 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:11.220508 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:11.220512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:11.220964 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:11.221167 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:11.221178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.222177 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.222181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:11.223176 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:11.223189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.223887 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.223892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:11.224046 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:11.224703 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:11.224721 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:11.228025 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":264,"threads_started":1}
W20260924 12:47:11.228506 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:11.219592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:38137 (local address 127.2.81.126:45011)
0924 12:47:11.219845 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:11.219851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:11.220125 (+ 274us) server_negotiation.cc:407] Connection header received
0924 12:47:11.220308 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:11.220312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:11.220386 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:11.220479 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:11.221311 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.222037 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.223330 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.223736 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.223846 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:11.228262 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:38137: BlockingRecv error: recv got EOF from 127.2.81.65:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 12:47:11 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:11 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:11.735680 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:11.721383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:39231)
0924 12:47:11.722118 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:11.722137 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:11.722271 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:11.722609 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:11.722614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:11.723046 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:11.723251 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:11.723264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.724566 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.724570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:11.725744 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:11.725760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.726548 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.726553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:11.726679 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:11.731394 (+ 4715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:11.731418 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:11.735393 (+ 3975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":375,"threads_started":1}
W20260924 12:47:11.736001 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:11.721537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:39231 (local address 127.2.81.126:45011)
0924 12:47:11.721731 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:11.721736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:11.722233 (+ 497us) server_negotiation.cc:407] Connection header received
0924 12:47:11.722373 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:11.722377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:11.722441 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:11.722532 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:11.723427 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.724420 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.725898 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:11.726379 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:11.726484 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:11.735555 (+ 9071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:39231: BlockingRecv error: recv got EOF from 127.2.81.66:39231 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 12:47:12 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:12 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:12.210736 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:12.201856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:33013)
0924 12:47:12.202441 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:12.202459 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:12.202604 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:12.202913 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:12.202916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:12.203329 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:12.203491 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:12.203500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.204565 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.204570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:12.205577 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:12.205592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.206397 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.206402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:12.206532 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:12.207236 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:12.207255 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:12.210380 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
W20260924 12:47:12.210870 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:12.201988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:33013 (local address 127.2.81.126:45011)
0924 12:47:12.202177 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:12.202182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:12.202555 (+ 373us) server_negotiation.cc:407] Connection header received
0924 12:47:12.202714 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:12.202718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:12.202781 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:12.202868 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:12.203647 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.204385 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.205740 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.206225 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.206335 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:12.210635 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:33013: BlockingRecv error: recv got EOF from 127.2.81.67:33013 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 12:47:12 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:12 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:12.238252 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:12.229445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:37799)
0924 12:47:12.229913 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:12.229932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:12.230050 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:12.230357 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:12.230360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:12.230736 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:12.230893 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:12.230904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.232053 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.232059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:12.233102 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:12.233116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.233948 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.233957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:12.234132 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:12.234875 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:12.234899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:12.238030 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
W20260924 12:47:12.238458 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:12.229593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:37799 (local address 127.2.81.126:45011)
0924 12:47:12.229760 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:12.229764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:12.230004 (+ 240us) server_negotiation.cc:407] Connection header received
0924 12:47:12.230156 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:12.230161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:12.230235 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:12.230320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:12.231056 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.231877 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.233255 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.233714 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.233831 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:12.238218 (+ 4387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:37799: BlockingRecv error: recv got EOF from 127.2.81.65:37799 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 12:47:12 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:12 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:12.744346 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:12.736596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:58067)
0924 12:47:12.737126 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:12.737154 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:12.737264 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:12.737521 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:12.737524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:12.737861 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:12.738009 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:12.738017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.739196 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.739202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:12.740137 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:12.740149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.740766 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.740770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:12.740873 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:12.741504 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:12.741522 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:12.744119 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":251,"threads_started":1}
W20260924 12:47:12.744510 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:12.736702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:58067 (local address 127.2.81.126:45011)
0924 12:47:12.736855 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:12.736860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:12.737226 (+ 366us) server_negotiation.cc:407] Connection header received
0924 12:47:12.737348 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:12.737351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:12.737414 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:12.737496 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:12.738167 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.739071 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.740245 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:12.740638 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:12.740724 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:12.744288 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:58067: BlockingRecv error: recv got EOF from 127.2.81.66:58067 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 12:47:13 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:13 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:13.220305 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:13.211780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:60557)
0924 12:47:13.212463 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:13.212481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:13.212622 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:13.212938 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:13.212944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:13.213358 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:13.213519 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:13.213529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.214554 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.214558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:13.215503 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:13.215517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.216253 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.216259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:13.216401 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:13.217145 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:13.217165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:13.219988 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":346,"threads_started":1}
W20260924 12:47:13.220582 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:13.211911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:60557 (local address 127.2.81.126:45011)
0924 12:47:13.212140 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:13.212153 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:13.212578 (+ 425us) server_negotiation.cc:407] Connection header received
0924 12:47:13.212735 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:13.212740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:13.212801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:13.212882 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:13.213667 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.214388 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.215646 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.216069 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.216180 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:13.220326 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:60557: BlockingRecv error: recv got EOF from 127.2.81.67:60557 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 12:47:13 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:13 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:13.246888 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:13.239047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:57947)
0924 12:47:13.239518 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:13.239534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:13.239644 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:13.239903 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:13.239906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:13.240264 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:13.240419 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:13.240457 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.241464 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.241468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:13.242373 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:13.242384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.243075 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.243080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:13.243203 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:13.243736 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:13.243751 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:13.246646 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
W20260924 12:47:13.247022 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:13.239120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:57947 (local address 127.2.81.126:45011)
0924 12:47:13.239328 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:13.239334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:13.239620 (+ 286us) server_negotiation.cc:407] Connection header received
0924 12:47:13.239737 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:13.239741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:13.239800 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:13.239877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:13.240547 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.241337 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.242531 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.242942 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.243040 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:13.246793 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:57947: BlockingRecv error: recv got EOF from 127.2.81.65:57947 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 12:47:13 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:13 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:13.754674 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:13.745368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:45251)
0924 12:47:13.745941 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:13.745961 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:13.746082 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:13.746525 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:13.746528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:13.746974 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:13.747172 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:13.747182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.748223 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.748227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:13.749418 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:13.749433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.750401 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.750407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:13.750627 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:13.751263 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:13.751283 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:13.754402 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":330,"threads_started":1}
W20260924 12:47:13.754889 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:13.745568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:45251 (local address 127.2.81.126:45011)
0924 12:47:13.745708 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:13.745713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:13.746059 (+ 346us) server_negotiation.cc:407] Connection header received
0924 12:47:13.746229 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:13.746234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:13.746301 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:13.746405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:13.747346 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.748084 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.749591 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:13.750229 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:13.750373 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:13.754667 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:45251: BlockingRecv error: recv got EOF from 127.2.81.66:45251 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 12:47:14 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:14 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:14.230679 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:14.221567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:49347)
0924 12:47:14.222115 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:14.222132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:14.222274 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:14.222781 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:14.222786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:14.223287 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:14.223497 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:14.223511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.224651 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.224655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:14.225703 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:14.225717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.226456 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.226461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:14.226599 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:14.227183 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:14.227202 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:14.230305 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":263,"threads_started":1}
W20260924 12:47:14.230772 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:14.221704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:49347 (local address 127.2.81.126:45011)
0924 12:47:14.221921 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:14.221928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:14.222222 (+ 294us) server_negotiation.cc:407] Connection header received
0924 12:47:14.222474 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:14.222480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:14.222561 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:14.222707 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:14.223649 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.224496 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.225864 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.226291 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.226396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:14.230563 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:49347: BlockingRecv error: recv got EOF from 127.2.81.67:49347 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 12:47:14 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:14 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:14.256313 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:14.247894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:34297)
0924 12:47:14.248484 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:14.248499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:14.248609 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:14.248957 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:14.248961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:14.249333 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:14.249512 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:14.249523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.250576 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.250580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:14.251593 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:14.251607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.252522 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.252528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:14.252659 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:14.253315 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:14.253333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:14.256064 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":368,"threads_started":1}
W20260924 12:47:14.256523 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:14.247929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:34297 (local address 127.2.81.126:45011)
0924 12:47:14.248101 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:14.248106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:14.248576 (+ 470us) server_negotiation.cc:407] Connection header received
0924 12:47:14.248723 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:14.248728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:14.248813 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:14.248894 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:14.249666 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.250401 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.251762 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.252279 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.252487 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:14.256286 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:34297: BlockingRecv error: recv got EOF from 127.2.81.65:34297 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 12:47:14 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:14 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:14.764178 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:14.755710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:39553)
0924 12:47:14.756137 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:14.756158 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:14.756279 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:14.756627 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:14.756631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:14.757062 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:14.757207 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:14.757217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.758312 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.758317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:14.759274 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:14.759292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.760004 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.760010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:14.760173 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:14.760828 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:14.760849 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:14.763950 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":235,"threads_started":1}
W20260924 12:47:14.764570 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:14.755850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:39553 (local address 127.2.81.126:45011)
0924 12:47:14.756089 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:14.756095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:14.756323 (+ 228us) server_negotiation.cc:407] Connection header received
0924 12:47:14.756386 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:14.756390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:14.756452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:14.756538 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:14.757350 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.758183 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.759412 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:14.759844 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:14.759975 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:14.764303 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:39553: BlockingRecv error: recv got EOF from 127.2.81.66:39553 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 12:47:15 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:15 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:15.240042 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:15.231582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:41483)
0924 12:47:15.232084 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:15.232100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:15.232220 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:15.232556 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:15.232561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:15.232948 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:15.233145 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:15.233155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.234114 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.234120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:15.235150 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:15.235163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.235942 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.235948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:15.236113 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:15.236684 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:15.236702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:15.239810 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":239,"threads_started":1}
W20260924 12:47:15.240348 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:15.231714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:41483 (local address 127.2.81.126:45011)
0924 12:47:15.231947 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:15.231953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:15.232171 (+ 218us) server_negotiation.cc:407] Connection header received
0924 12:47:15.232359 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:15.232364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:15.232432 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:15.232514 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:15.233274 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.233979 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.235287 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.235788 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.235910 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:15.240097 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:41483: BlockingRecv error: recv got EOF from 127.2.81.67:41483 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 24 12:47:15 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:15 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:15.266500 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:15.257396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:49417)
0924 12:47:15.257929 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:15.257946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:15.258058 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:15.258365 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:15.258369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:15.258766 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:15.258934 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:15.258943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.260133 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.260137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:15.261299 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:15.261312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.262068 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.262073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:15.262262 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:15.262893 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:15.262912 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:15.266102 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":308,"threads_started":1}
W20260924 12:47:15.266749 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:15.257452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:49417 (local address 127.2.81.126:45011)
0924 12:47:15.257663 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:15.257667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:15.258040 (+ 373us) server_negotiation.cc:407] Connection header received
0924 12:47:15.258163 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:15.258169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:15.258247 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:15.258329 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:15.259095 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.259910 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.261444 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.261896 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.262000 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:15.266357 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:49417: BlockingRecv error: recv got EOF from 127.2.81.65:49417 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 12:47:15 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:15 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:15.774878 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:15.765369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:42295)
0924 12:47:15.765848 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:15.765869 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:15.765996 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:15.766528 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:15.766533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:15.767004 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:15.767235 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:15.767247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.768613 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.768618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:15.769627 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:15.769641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.770491 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.770496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:15.770633 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:15.771501 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:15.771520 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:15.774607 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
W20260924 12:47:15.775102 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:15.765546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:42295 (local address 127.2.81.126:45011)
0924 12:47:15.765767 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:15.765774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:15.766027 (+ 253us) server_negotiation.cc:407] Connection header received
0924 12:47:15.766174 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:15.766180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:15.766275 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:15.766388 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:15.767366 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.768430 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.769784 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:15.770224 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:15.770338 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:15.774867 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:42295: BlockingRecv error: recv got EOF from 127.2.81.66:42295 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 12:47:16 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:16 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:16.249224 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:16.241128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:43897)
0924 12:47:16.241614 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:16.241630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:16.241739 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:16.242074 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:16.242078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:16.242477 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:16.242679 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:16.242688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.243795 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.243800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:16.244774 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:16.244787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.245474 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.245486 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:16.245618 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:16.246206 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:16.246224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:16.248951 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
W20260924 12:47:16.249477 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:16.241190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:43897 (local address 127.2.81.126:45011)
0924 12:47:16.241335 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:16.241340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:16.241703 (+ 363us) server_negotiation.cc:407] Connection header received
0924 12:47:16.241875 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:16.241880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:16.241943 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:16.242058 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:16.242812 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.243594 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.244892 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.245317 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.245411 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:16.249255 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:43897: BlockingRecv error: recv got EOF from 127.2.81.67:43897 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 12:47:16 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:16 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:16.275931 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:16.267435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:50261)
0924 12:47:16.267994 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:16.268010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:16.268120 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:16.268522 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:16.268526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:16.268926 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:16.269129 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:16.269140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.270223 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.270227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:16.271198 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:16.271212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.272088 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.272093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:16.272237 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:16.272849 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:16.272868 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:16.275648 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":325,"threads_started":1}
W20260924 12:47:16.276135 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:16.267549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:50261 (local address 127.2.81.126:45011)
0924 12:47:16.267757 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:16.267762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:16.268109 (+ 347us) server_negotiation.cc:407] Connection header received
0924 12:47:16.268263 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:16.268267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:16.268329 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:16.268418 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:16.269326 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.270074 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.271356 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.271835 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.272042 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:16.275864 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:50261: BlockingRecv error: recv got EOF from 127.2.81.65:50261 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 12:47:16 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:16 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:16.788486 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:16.775820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:35971)
0924 12:47:16.776379 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:16.776399 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:16.776540 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:16.776898 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:16.776903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:16.777560 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:16.777734 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:16.777745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.778796 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.778801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:16.779727 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:16.779740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.780478 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.780484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:16.780662 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:16.785167 (+ 4505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:16.785191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:16.788151 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":326,"threads_started":1}
W20260924 12:47:16.788590 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:16.775909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:35971 (local address 127.2.81.126:45011)
0924 12:47:16.776196 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:16.776204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:16.776490 (+ 286us) server_negotiation.cc:407] Connection header received
0924 12:47:16.776688 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:16.776692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:16.776753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:16.776836 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:16.777927 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.778643 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.779890 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:16.780323 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:16.780437 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:16.788379 (+ 7942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:35971: BlockingRecv error: recv got EOF from 127.2.81.66:35971 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 12:47:17 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:17 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:17.262303 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:17.250336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:36003)
0924 12:47:17.250857 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:17.250874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:17.251044 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:17.251734 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:17.251738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:17.252306 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:17.252500 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:17.252510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.253971 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.253978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:17.255421 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:17.255442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.256644 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.256651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:17.256794 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:17.257674 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:17.257704 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:17.262037 (+ 4333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":298,"threads_started":1}
W20260924 12:47:17.262799 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:17.250488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:36003 (local address 127.2.81.126:45011)
0924 12:47:17.250706 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:17.250712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:17.250977 (+ 265us) server_negotiation.cc:407] Connection header received
0924 12:47:17.251435 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:17.251440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:17.251512 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:17.251662 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:17.252750 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.253636 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.255617 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.256434 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.256592 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:17.262482 (+ 5890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:36003: BlockingRecv error: recv got EOF from 127.2.81.67:36003 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 12:47:17 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:17 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:17.286141 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:17.277162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:38971)
0924 12:47:17.277659 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:17.277675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:17.277809 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:17.278299 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:17.278303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:17.278762 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:17.278953 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:17.278964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.280259 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.280264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:17.281414 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:17.281430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.282293 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.282298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:17.282427 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:17.283056 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:17.283075 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:17.285896 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":255,"threads_started":1}
W20260924 12:47:17.286402 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:17.277310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:38971 (local address 127.2.81.126:45011)
0924 12:47:17.277525 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:17.277532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:17.277837 (+ 305us) server_negotiation.cc:407] Connection header received
0924 12:47:17.277936 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:17.277940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:17.278006 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:17.278170 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:17.279144 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.280077 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.281563 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.282062 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.282175 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:17.286145 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:38971: BlockingRecv error: recv got EOF from 127.2.81.65:38971 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 12:47:17 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:17 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:17.798635 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:17.789630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:58453)
0924 12:47:17.790171 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:17.790194 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:17.790311 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:17.790658 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:17.790662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:17.791033 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:17.791198 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:17.791207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.792450 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.792455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:17.793788 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:17.793806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.794504 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.794511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:17.794686 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:17.795364 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:17.795382 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:17.798378 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":249,"threads_started":1}
W20260924 12:47:17.798964 11671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:17.789886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:58453 (local address 127.2.81.126:45011)
0924 12:47:17.790380 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:17.790385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:17.790401 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:47:17.790454 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:17.790458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:17.790516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:17.790606 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:17.791365 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.792299 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.793916 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:17.794330 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:17.794440 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:17.798681 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:58453: BlockingRecv error: recv got EOF from 127.2.81.66:58453 (error 108)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":305,"threads_started":1}
Sep 24 12:47:18 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:18 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:18.272641 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:18.263428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:39391)
0924 12:47:18.264002 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:18.264018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:18.264146 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:18.264482 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:18.264486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:18.264870 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:18.265126 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:18.265138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.266295 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.266300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:18.267239 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:18.267251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.268027 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.268032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:18.268193 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:18.268822 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:18.268843 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:18.272380 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":322,"threads_started":1}
W20260924 12:47:18.272946 11671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:18.263509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:39391 (local address 127.2.81.126:45011)
0924 12:47:18.263700 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:18.263705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:18.264106 (+ 401us) server_negotiation.cc:407] Connection header received
0924 12:47:18.264262 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:18.264268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:18.264343 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:18.264429 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:18.265311 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.266145 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.267400 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.267828 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.267935 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:18.272692 (+ 4757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:39391: BlockingRecv error: recv got EOF from 127.2.81.67:39391 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 12:47:18 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:18 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:18.298162 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:18.287858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:48031)
0924 12:47:18.288391 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:18.288410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:18.288565 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:18.289041 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:18.289046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:18.289492 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:18.289725 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:18.289736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.290791 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.290797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:18.291704 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:18.291717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.292456 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.292462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:18.292638 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:18.293620 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:18.293643 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:18.297722 (+ 4079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
W20260924 12:47:18.298354 11671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:18.287954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:48031 (local address 127.2.81.126:45011)
0924 12:47:18.288285 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:18.288291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:18.288539 (+ 248us) server_negotiation.cc:407] Connection header received
0924 12:47:18.288710 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:18.288716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:18.288802 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:18.288921 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:18.289891 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.290637 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.291855 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.292278 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.292416 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:18.298092 (+ 5676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:48031: BlockingRecv error: recv got EOF from 127.2.81.65:48031 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Sep 24 12:47:18 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:18 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:18.808257 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:18.799745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:47145)
0924 12:47:18.800261 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:18.800286 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:18.800398 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:18.800870 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:18.800875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:18.801290 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:18.801457 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:18.801467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.802483 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.802487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:18.803379 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:18.803392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.804100 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.804114 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:18.804273 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:18.805116 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:18.805136 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:18.807860 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":218,"threads_started":1}
W20260924 12:47:18.808320 11675 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:18.799836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:47145 (local address 127.2.81.126:45011)
0924 12:47:18.800530 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:18.800535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:18.800551 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:47:18.800598 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:18.800603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:18.800715 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:18.800815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:18.801627 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.802342 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.803516 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:18.803957 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:18.804065 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:18.808111 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:47145: BlockingRecv error: recv got EOF from 127.2.81.66:47145 (error 108)
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":483,"threads_started":1}
Sep 24 12:47:19 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:19 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.67@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:19.283069 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:19.273740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.67:41069)
0924 12:47:19.274168 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:19.274189 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:19.274375 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:19.274751 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:19.274754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:19.275087 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:19.275280 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:19.275288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.276567 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.276572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:19.277829 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:19.277846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.278622 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.278627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:19.278754 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:19.279499 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:19.279524 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:19.282812 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
W20260924 12:47:19.283357 11675 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:19.273916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:41069 (local address 127.2.81.126:45011)
0924 12:47:19.274174 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:19.274180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:19.274300 (+ 120us) server_negotiation.cc:407] Connection header received
0924 12:47:19.274514 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:19.274518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:19.274589 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:19.274678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:19.275527 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.276389 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.277992 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.278461 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.278582 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:19.283122 (+ 4540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.67:41069: BlockingRecv error: recv got EOF from 127.2.81.67:41069 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 24 12:47:19 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:19 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.65@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:19.308233 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:19.299779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.65:34263)
0924 12:47:19.300339 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:19.300355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:19.300492 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:19.300837 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:19.300842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:19.301296 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:19.301490 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:19.301500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.302534 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.302538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:19.303519 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:19.303533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.304224 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.304230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:19.304382 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:19.304955 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:19.305013 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:19.307961 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":348,"threads_started":1}
W20260924 12:47:19.308432 11675 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:19.299932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:34263 (local address 127.2.81.126:45011)
0924 12:47:19.300176 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:19.300182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:19.300439 (+ 257us) server_negotiation.cc:407] Connection header received
0924 12:47:19.300619 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:19.300624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:19.300695 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:19.300776 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:19.301650 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.302382 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.303673 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.304076 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.304187 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:19.308179 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.65:34263: BlockingRecv error: recv got EOF from 127.2.81.65:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 12:47:19 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 12:47:19 dist-test-slave-9tm1 krb5kdc[10629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.81.66@KRBTEST.COM for oryx/127.2.81.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 12:47:19.820314 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:19.809829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45011 (local address 127.2.81.66:55793)
0924 12:47:19.810398 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:19.810426 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:19.810842 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:19.811350 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:19.811355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:19.811909 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:19.812128 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:19.812138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.813538 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.813558 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:19.814541 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:19.814556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.815546 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.815551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:19.815681 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:19.816328 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:19.816348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:19.819948 (+ 3600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.81.126:45011: Server oryx/127.2.81.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":308,"threads_started":1}
W20260924 12:47:19.820524 11679 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:19.810000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:55793 (local address 127.2.81.126:45011)
0924 12:47:19.810711 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:19.810716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:19.810733 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:47:19.811061 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:19.811067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:19.811168 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:19.811265 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:19.812300 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.813136 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.814716 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:19.815297 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:19.815520 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:19.820273 (+ 4753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.81.66:55793: BlockingRecv error: recv got EOF from 127.2.81.66:55793 (error 108)
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":509,"threads_started":1}
I20260924 12:47:20.113425 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11217
I20260924 12:47:20.127599 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:20.114506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.2.81.65:33449)
0924 12:47:20.115073 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:20.115081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:20.115108 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:47:20.115185 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:20.115191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:20.115407 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:20.115567 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:20.117804 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.118862 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.120495 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.121206 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.121332 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:20.123968 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:20.123983 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:20.123999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:20.124040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:20.127044 (+ 3004us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.81.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 12:47:20.127367 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42736: Request ticket server kudu/127.2.81.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
W20260924 12:47:20.127688 11681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42736: Request ticket server kudu/127.2.81.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 12:47:20.128028 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11217: Not authorized: Client connection negotiation failed: client connection to 127.2.81.65:33449: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.81.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 12:47:20.128237 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11217
I20260924 12:47:20.142108 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11369
I20260924 12:47:20.154723 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:20.143290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.2.81.66:34363)
0924 12:47:20.143929 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:20.143936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:20.143973 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:47:20.144083 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:20.144090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:20.144313 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:20.144481 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:20.145882 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.147094 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.148320 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.148906 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.149060 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:20.151567 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:20.151576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:20.151590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:20.151629 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:20.154206 (+ 2577us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.81.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 12:47:20.154487 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37052: Request ticket server kudu/127.2.81.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":314,"threads_started":1}
W20260924 12:47:20.154822 11682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37052: Request ticket server kudu/127.2.81.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 12:47:20.154973 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11369: Not authorized: Client connection negotiation failed: client connection to 127.2.81.66:34363: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.81.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 12:47:20.155118 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11369
I20260924 12:47:20.168725 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11500
I20260924 12:47:20.181254 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:20.169817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48452 (local address 127.2.81.67:42677)
0924 12:47:20.170436 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:20.170443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:20.170467 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:47:20.170525 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:20.170532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:20.170755 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:20.170902 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:20.172230 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.173364 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.174572 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.175109 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.175212 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:20.177955 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:20.177973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:20.177988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:20.178032 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:20.180654 (+ 2622us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.81.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 12:47:20.180942 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48452: Request ticket server kudu/127.2.81.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":358,"threads_started":1}
W20260924 12:47:20.181370 11683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48452: Request ticket server kudu/127.2.81.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 12:47:20.181533 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11500: Not authorized: Client connection negotiation failed: client connection to 127.2.81.67:42677: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.81.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 12:47:20.181704 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11500
I20260924 12:47:20.193825 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11148
I20260924 12:47:20.195055 11210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:20.310593 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11148
2026-09-24T12:47:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21239 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T12:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:47:20Z Disabled control of system clock
I20260924 12:47:20.380697 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37185
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:37185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 12:47:20.712750 11688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:20.713093 11688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:20.713172 11688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:20.721009 11688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:47:20.721143 11688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:20.721180 11688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:20.721236 11688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:47:20.721279 11688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:47:20.730929 11688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:37185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:20.732256 11688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:20.734094 11688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:20.747071 11693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:20.747118 11694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:20.747030 11696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:20.750846 11688 server_base.cc:1034] running on GCE node
I20260924 12:47:20.751775 11688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:20.753316 11688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:20.754528 11688 hybrid_clock.cc:648] HybridClock initialized: now 1790254040754474 us; error 55 us; skew 500 ppm
I20260924 12:47:20.754884 11688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:20.757498 11688 webserver.cc:460] Webserver started at http://127.2.81.126:39523/ using document root <none> and password file <none>
I20260924 12:47:20.758255 11688 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:20.758363 11688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:20.758675 11688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:20.761751 11688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "9b75c2390c534b899d82b2e5085b9c28"
format_stamp: "Formatted at 2026-09-24 12:47:20 on dist-test-slave-9tm1"
server_key: "e3a0b73c64ac8ab79a6e1004c3655d6f"
server_key_iv: "d400dacf1d73c04fb17c0f158f663fbe"
server_key_version: "encryptionkey@0"
I20260924 12:47:20.762600 11688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "9b75c2390c534b899d82b2e5085b9c28"
format_stamp: "Formatted at 2026-09-24 12:47:20 on dist-test-slave-9tm1"
server_key: "e3a0b73c64ac8ab79a6e1004c3655d6f"
server_key_iv: "d400dacf1d73c04fb17c0f158f663fbe"
server_key_version: "encryptionkey@0"
I20260924 12:47:20.768421 11688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:47:20.772934 11702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:20.774777 11688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:47:20.775059 11688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "9b75c2390c534b899d82b2e5085b9c28"
format_stamp: "Formatted at 2026-09-24 12:47:20 on dist-test-slave-9tm1"
server_key: "e3a0b73c64ac8ab79a6e1004c3655d6f"
server_key_iv: "d400dacf1d73c04fb17c0f158f663fbe"
server_key_version: "encryptionkey@0"
I20260924 12:47:20.775295 11688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:20.793543 11688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:20.794426 11688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:20.794696 11688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:20.812364 11688 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:37185
I20260924 12:47:20.812387 11753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:37185 every 8 connection(s)
I20260924 12:47:20.813762 11688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:47:20.817755 11754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:20.822901 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11688
I20260924 12:47:20.823060 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:47:20.823663 2373 external_mini_cluster.cc:1428] Setting key c98a9d164e86a09db0443a2ee94f7745
I20260924 12:47:20.828665 11754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28: Bootstrap starting.
I20260924 12:47:20.832305 11754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:20.833583 11754 log.cc:824] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:20.836709 11754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28: No bootstrap required, opened a new log
I20260924 12:47:20.837643 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:20.825714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.2.81.126:37185)
0924 12:47:20.826762 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:20.826774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:20.826816 (+ 42us) server_negotiation.cc:407] Connection header received
0924 12:47:20.828456 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:20.828478 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:20.829032 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:20.829385 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:20.830018 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.831441 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.833186 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:20.833923 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:20.834175 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:20.835235 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:20.835243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:20.835259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:20.835787 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:20.835866 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:20.835887 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:20.835973 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:20.836260 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 12:47:20.836576 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":370,"threads_started":1}
I20260924 12:47:20.845592 11754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75c2390c534b899d82b2e5085b9c28" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37185 } }
I20260924 12:47:20.846032 11754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:20.846144 11754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b75c2390c534b899d82b2e5085b9c28, State: Initialized, Role: FOLLOWER
I20260924 12:47:20.846709 11754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75c2390c534b899d82b2e5085b9c28" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37185 } }
I20260924 12:47:20.846931 11754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:47:20.847051 11754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:47:20.847203 11754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:20.849391 11754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75c2390c534b899d82b2e5085b9c28" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37185 } }
I20260924 12:47:20.849901 11754 leader_election.cc:302] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b75c2390c534b899d82b2e5085b9c28; no voters:
I20260924 12:47:20.850478 11754 leader_election.cc:288] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:47:20.850646 11759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:20.851303 11759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [term 1 LEADER]: Becoming Leader. State: Replica: 9b75c2390c534b899d82b2e5085b9c28, State: Running, Role: LEADER
I20260924 12:47:20.851930 11759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75c2390c534b899d82b2e5085b9c28" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37185 } }
I20260924 12:47:20.852768 11754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:47:20.854650 11760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b75c2390c534b899d82b2e5085b9c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75c2390c534b899d82b2e5085b9c28" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37185 } } }
I20260924 12:47:20.855194 11760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:20.855417 11761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b75c2390c534b899d82b2e5085b9c28. Latest consensus state: current_term: 1 leader_uuid: "9b75c2390c534b899d82b2e5085b9c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b75c2390c534b899d82b2e5085b9c28" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37185 } } }
I20260924 12:47:20.856274 11761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28 [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:20.857002 11764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:47:20.862870 11764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:47:20.872925 11764 catalog_manager.cc:1347] Generated new cluster ID: b0f8e31c38d6445288688bd9086f011a
I20260924 12:47:20.873083 11764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:47:20.885949 11764 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:47:20.887825 11764 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:47:20.897938 11764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b75c2390c534b899d82b2e5085b9c28: Generated new TSK 0
I20260924 12:47:20.899035 11764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:47:20.909525 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37185
--builtin_ntp_servers=127.2.81.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 12:47:21.255540 11778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:21.255823 11778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:21.255924 11778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:21.263350 11778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:21.263456 11778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:21.263588 11778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:47:21.273469 11778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:21.274760 11778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:21.276522 11778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:21.289796 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:21.289850 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:21.289703 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:21.290462 11778 server_base.cc:1034] running on GCE node
I20260924 12:47:21.291086 11778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:21.292076 11778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:21.293309 11778 hybrid_clock.cc:648] HybridClock initialized: now 1790254041293239 us; error 56 us; skew 500 ppm
I20260924 12:47:21.293720 11778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:21.296295 11778 webserver.cc:460] Webserver started at http://127.2.81.65:34739/ using document root <none> and password file <none>
I20260924 12:47:21.297042 11778 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:21.297161 11778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:21.297466 11778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:21.300303 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "bd3570b8a25a4dcebe234e10681575fc"
format_stamp: "Formatted at 2026-09-24 12:47:21 on dist-test-slave-9tm1"
server_key: "b275ca18fbc358df9b2b0c65ae3686ea"
server_key_iv: "a08826ccd8c590283f0b608710c71836"
server_key_version: "encryptionkey@0"
I20260924 12:47:21.301126 11778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "bd3570b8a25a4dcebe234e10681575fc"
format_stamp: "Formatted at 2026-09-24 12:47:21 on dist-test-slave-9tm1"
server_key: "b275ca18fbc358df9b2b0c65ae3686ea"
server_key_iv: "a08826ccd8c590283f0b608710c71836"
server_key_version: "encryptionkey@0"
I20260924 12:47:21.306252 11778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 12:47:21.309830 11792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:21.311234 11778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:47:21.311390 11778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "bd3570b8a25a4dcebe234e10681575fc"
format_stamp: "Formatted at 2026-09-24 12:47:21 on dist-test-slave-9tm1"
server_key: "b275ca18fbc358df9b2b0c65ae3686ea"
server_key_iv: "a08826ccd8c590283f0b608710c71836"
server_key_version: "encryptionkey@0"
I20260924 12:47:21.311589 11778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:21.329697 11778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:21.330606 11778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:21.330883 11778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:21.331857 11778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:21.333412 11778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:21.333523 11778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:21.333632 11778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:21.333701 11778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:21.365226 11778 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:46001
I20260924 12:47:21.365257 11904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:46001 every 8 connection(s)
I20260924 12:47:21.366499 11778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:47:21.375635 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11778
I20260924 12:47:21.375829 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:47:21.376423 2373 external_mini_cluster.cc:1428] Setting key 985fe032d1e972f5b101264f841cacc0
I20260924 12:47:21.379441 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37185
--builtin_ntp_servers=127.2.81.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 12:47:21.381215 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:21.369497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:56417 (local address 127.2.81.126:37185)
0924 12:47:21.370044 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:21.370049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:21.371506 (+ 1457us) server_negotiation.cc:407] Connection header received
0924 12:47:21.373344 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:21.373350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:21.373442 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:21.373543 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:21.374725 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.375920 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.377871 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.378482 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.378611 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:21.380249 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:21.380256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:21.380261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:21.380289 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:21.380342 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:21.380357 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:21.380449 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:21.380733 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 12:47:21.380832 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":332,"threads_started":1}
I20260924 12:47:21.382282 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:21.370292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37185 (local address 127.2.81.65:56417)
0924 12:47:21.371328 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:21.371367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:21.373020 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:21.373810 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:21.373819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:21.373920 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:21.374547 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:21.374567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.376097 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.376101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:21.377694 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:21.377719 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.378708 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.378714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:21.378944 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:21.379858 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:47:21.379861 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:21.379914 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:47:21.380021 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:21.380517 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:21.380528 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:21.380536 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:47:21.380715 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 12:47:21.381121 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":315,"threads_started":1}
I20260924 12:47:21.384007 11905 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37185
I20260924 12:47:21.384336 11905 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:21.385200 11905 heartbeater.cc:511] Master 127.2.81.126:37185 requested a full tablet report, sending...
I20260924 12:47:21.387477 11719 ts_manager.cc:194] Registered new tserver with Master: bd3570b8a25a4dcebe234e10681575fc (127.2.81.65:46001)
I20260924 12:47:21.389024 11719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.65:56417
W20260924 12:47:21.713747 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:21.714044 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:21.714169 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:21.721604 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:21.721711 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:21.721848 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:47:21.731310 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:21.732569 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:21.734503 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:21.747699 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:21.747673 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:21.747735 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:21.748021 11909 server_base.cc:1034] running on GCE node
I20260924 12:47:21.748888 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:21.749949 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:21.751169 11909 hybrid_clock.cc:648] HybridClock initialized: now 1790254041751114 us; error 39 us; skew 500 ppm
I20260924 12:47:21.751518 11909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:21.754092 11909 webserver.cc:460] Webserver started at http://127.2.81.66:44691/ using document root <none> and password file <none>
I20260924 12:47:21.754884 11909 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:21.754979 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:21.755257 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:21.758425 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "d716115c9c504e5da97fa09075895d8a"
format_stamp: "Formatted at 2026-09-24 12:47:21 on dist-test-slave-9tm1"
server_key: "aaaf58b49fd444dc0f6492bcb3e86980"
server_key_iv: "728990e3fc0aaf83a37b74912e8a30a6"
server_key_version: "encryptionkey@0"
I20260924 12:47:21.759274 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "d716115c9c504e5da97fa09075895d8a"
format_stamp: "Formatted at 2026-09-24 12:47:21 on dist-test-slave-9tm1"
server_key: "aaaf58b49fd444dc0f6492bcb3e86980"
server_key_iv: "728990e3fc0aaf83a37b74912e8a30a6"
server_key_version: "encryptionkey@0"
I20260924 12:47:21.765173 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 12:47:21.769065 11923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:21.770607 11909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:47:21.770771 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "d716115c9c504e5da97fa09075895d8a"
format_stamp: "Formatted at 2026-09-24 12:47:21 on dist-test-slave-9tm1"
server_key: "aaaf58b49fd444dc0f6492bcb3e86980"
server_key_iv: "728990e3fc0aaf83a37b74912e8a30a6"
server_key_version: "encryptionkey@0"
I20260924 12:47:21.770965 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:21.785944 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:21.786808 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:21.787045 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:21.788096 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:21.789702 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:21.789789 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:21.789872 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:21.789927 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:21.819608 11909 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:39733
I20260924 12:47:21.819621 12035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:39733 every 8 connection(s)
I20260924 12:47:21.821187 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:47:21.823570 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 11909
I20260924 12:47:21.823704 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:47:21.824252 2373 external_mini_cluster.cc:1428] Setting key 8085729eb5fe6ef6254eb89699c243aa
I20260924 12:47:21.827241 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37185
--builtin_ntp_servers=127.2.81.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 12:47:21.842123 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:21.823957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44915 (local address 127.2.81.126:37185)
0924 12:47:21.824159 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:21.824165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:21.826185 (+ 2020us) server_negotiation.cc:407] Connection header received
0924 12:47:21.828536 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:21.828549 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:21.828631 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:21.828744 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:21.830166 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.831117 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.833245 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.833658 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.833762 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:21.841207 (+ 7445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:21.841215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:21.841220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:21.841246 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:21.841288 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:21.841306 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:21.841401 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:21.841700 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 12:47:21.841815 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:47:21.843187 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:21.824538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37185 (local address 127.2.81.66:44915)
0924 12:47:21.826003 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:21.826060 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:21.828210 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:21.829051 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:21.829063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:21.829202 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:21.829980 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:21.830004 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.831266 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.831271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:21.833085 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:21.833100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:21.833877 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:21.833883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:21.834169 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:21.840716 (+ 6547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:47:21.840719 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:21.840773 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:47:21.840878 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:21.841445 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:21.841456 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:21.841465 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:47:21.841648 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 12:47:21.842007 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":616,"threads_started":1}
I20260924 12:47:21.844762 12036 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37185
I20260924 12:47:21.845095 12036 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:21.845882 12036 heartbeater.cc:511] Master 127.2.81.126:37185 requested a full tablet report, sending...
I20260924 12:47:21.847354 11719 ts_manager.cc:194] Registered new tserver with Master: d716115c9c504e5da97fa09075895d8a (127.2.81.66:39733)
I20260924 12:47:21.848165 11719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.66:44915
W20260924 12:47:22.179829 12039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:22.180379 12039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:22.180539 12039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:22.188398 12039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:22.188550 12039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:22.188687 12039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:47:22.198973 12039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:22.200403 12039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:22.202343 12039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:22.215961 12045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:22.215927 12044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:22.216040 12047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:22.216534 12039 server_base.cc:1034] running on GCE node
I20260924 12:47:22.217329 12039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:22.218372 12039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:22.219614 12039 hybrid_clock.cc:648] HybridClock initialized: now 1790254042219549 us; error 56 us; skew 500 ppm
I20260924 12:47:22.220000 12039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:22.222541 12039 webserver.cc:460] Webserver started at http://127.2.81.67:35255/ using document root <none> and password file <none>
I20260924 12:47:22.223256 12039 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:22.223349 12039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:22.223675 12039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:22.226663 12039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "7dff631277aa429fa19258fcaee51644"
format_stamp: "Formatted at 2026-09-24 12:47:22 on dist-test-slave-9tm1"
server_key: "a09a2053b64df74d590d4b07b46df222"
server_key_iv: "f2c759cebba82b03c099c382478512cc"
server_key_version: "encryptionkey@0"
I20260924 12:47:22.227434 12039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "7dff631277aa429fa19258fcaee51644"
format_stamp: "Formatted at 2026-09-24 12:47:22 on dist-test-slave-9tm1"
server_key: "a09a2053b64df74d590d4b07b46df222"
server_key_iv: "f2c759cebba82b03c099c382478512cc"
server_key_version: "encryptionkey@0"
I20260924 12:47:22.233156 12039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:47:22.236774 12053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:22.238551 12039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:47:22.238745 12039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "7dff631277aa429fa19258fcaee51644"
format_stamp: "Formatted at 2026-09-24 12:47:22 on dist-test-slave-9tm1"
server_key: "a09a2053b64df74d590d4b07b46df222"
server_key_iv: "f2c759cebba82b03c099c382478512cc"
server_key_version: "encryptionkey@0"
I20260924 12:47:22.238965 12039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:22.256022 12039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:22.256880 12039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:22.257169 12039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:22.258361 12039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:22.259974 12039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:22.260070 12039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:22.260190 12039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:22.260263 12039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:22.290488 12039 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:33681
I20260924 12:47:22.290539 12165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:33681 every 8 connection(s)
I20260924 12:47:22.291764 12039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:47:22.301791 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12039
I20260924 12:47:22.301972 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:47:22.302471 2373 external_mini_cluster.cc:1428] Setting key 8ab00a799c67dd677327612d9e47d808
I20260924 12:47:22.304425 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:22.294327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:59303 (local address 127.2.81.126:37185)
0924 12:47:22.294502 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:22.294507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:22.296026 (+ 1519us) server_negotiation.cc:407] Connection header received
0924 12:47:22.297811 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:22.297816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:22.297885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:22.297974 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:22.299003 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.299780 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.301393 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.302004 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.302141 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:22.303579 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:22.303586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:22.303590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:22.303614 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:22.303651 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:22.303667 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:22.303761 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:22.304007 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 12:47:22.304098 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:47:22.305649 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:22.294795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37185 (local address 127.2.81.67:59303)
0924 12:47:22.295841 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:22.295882 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:22.297558 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:22.298169 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:22.298177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:22.298266 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:22.298860 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:22.298877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.299927 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.299931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:22.301226 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:22.301240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.302249 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.302255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:22.302475 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:22.303250 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:47:22.303253 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:22.303301 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:47:22.303382 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:22.303776 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:22.303787 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:22.303795 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:47:22.304012 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 12:47:22.304411 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":375,"threads_started":1}
I20260924 12:47:22.306957 12166 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37185
I20260924 12:47:22.307235 12166 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:22.307987 12166 heartbeater.cc:511] Master 127.2.81.126:37185 requested a full tablet report, sending...
I20260924 12:47:22.309338 11719 ts_manager.cc:194] Registered new tserver with Master: 7dff631277aa429fa19258fcaee51644 (127.2.81.67:33681)
I20260924 12:47:22.310117 11719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.67:59303
I20260924 12:47:22.317862 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 12:47:22.323704 11907 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:22.322209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.2.81.126:37185)
0924 12:47:22.322459 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:22.322465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:22.322684 (+ 219us) server_negotiation.cc:407] Connection header received
0924 12:47:22.322791 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:22.322796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:22.322873 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:22.322955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:22.323465 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52586: BlockingRecv error: recv got EOF from 127.0.0.1:52586 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:47:22.324645 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11778
I20260924 12:47:22.332497 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:22.325631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.2.81.65:46001)
0924 12:47:22.326121 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:22.326128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:22.326153 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:47:22.326206 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:22.326213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:22.326420 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:22.326544 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:22.326908 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.327899 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.329205 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.329771 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.329874 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:22.330971 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:22.330979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:22.330994 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:22.331632 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:22.331686 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:22.331706 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:22.331798 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:22.331963 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 12:47:22.332111 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260924 12:47:22.333791 11900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:22.414446 11905 heartbeater.cc:503] Master 127.2.81.126:37185 was elected leader, sending a full tablet report...
I20260924 12:47:22.415614 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11778
W20260924 12:47:22.429560 11706 connection.cc:535] server connection from 127.2.81.65:56417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:47:22.429852 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11909
I20260924 12:47:22.437453 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:22.430683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.2.81.66:39733)
0924 12:47:22.431137 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:22.431173 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:22.431198 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:47:22.431254 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:22.431260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:22.431483 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:22.431627 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:22.432003 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.432959 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.434176 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.434769 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.434899 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:22.435935 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:22.435943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:22.435957 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:22.436583 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:22.436631 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:22.436653 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:22.436767 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:22.436914 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 12:47:22.437122 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":206,"threads_started":1}
I20260924 12:47:22.438884 12031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:22.520644 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11909
I20260924 12:47:22.533687 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12039
I20260924 12:47:22.543459 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:22.534616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.2.81.67:33681)
0924 12:47:22.535287 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:22.535295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:22.535328 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:47:22.535409 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:22.535417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:22.535740 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:22.535925 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:22.536445 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.537594 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.538860 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:22.539512 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:22.539630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:22.541081 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:22.541093 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:22.541109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:22.541737 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:22.541796 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:22.541817 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:22.541921 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:22.542575 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 12:47:22.542841 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":356,"threads_started":1}
I20260924 12:47:22.545671 12161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:22.633276 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12039
I20260924 12:47:22.646070 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 11688
I20260924 12:47:22.647523 11749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:22.751924 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 11688
2026-09-24T12:47:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2437 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T12:47:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:47:22Z Disabled control of system clock
I20260924 12:47:22.818064 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44251
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:44251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 12:47:23.146456 12185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:23.146768 12185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:23.146852 12185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:23.154254 12185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:47:23.154376 12185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:23.154412 12185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:23.154458 12185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:47:23.154500 12185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:47:23.163803 12185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:44251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:23.165038 12185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:23.166800 12185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:23.178613 12191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:23.178539 12193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:23.178579 12190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:23.179854 12185 server_base.cc:1034] running on GCE node
I20260924 12:47:23.180692 12185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:23.182302 12185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:23.183538 12185 hybrid_clock.cc:648] HybridClock initialized: now 1790254043183495 us; error 41 us; skew 500 ppm
I20260924 12:47:23.183905 12185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:23.186549 12185 webserver.cc:460] Webserver started at http://127.2.81.126:38273/ using document root <none> and password file <none>
I20260924 12:47:23.187295 12185 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:23.187405 12185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:23.187733 12185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:23.192082 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "3254bdf486cc4cf0acec6de5561e04ef"
format_stamp: "Formatted at 2026-09-24 12:47:23 on dist-test-slave-9tm1"
server_key: "e122f5ca734c4b9834da13ace03d88c8"
server_key_iv: "8db1c09cf4f0f88da26050a95ade12a5"
server_key_version: "encryptionkey@0"
I20260924 12:47:23.193137 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "3254bdf486cc4cf0acec6de5561e04ef"
format_stamp: "Formatted at 2026-09-24 12:47:23 on dist-test-slave-9tm1"
server_key: "e122f5ca734c4b9834da13ace03d88c8"
server_key_iv: "8db1c09cf4f0f88da26050a95ade12a5"
server_key_version: "encryptionkey@0"
I20260924 12:47:23.199468 12185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 12:47:23.203441 12199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:23.205135 12185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:47:23.205322 12185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "3254bdf486cc4cf0acec6de5561e04ef"
format_stamp: "Formatted at 2026-09-24 12:47:23 on dist-test-slave-9tm1"
server_key: "e122f5ca734c4b9834da13ace03d88c8"
server_key_iv: "8db1c09cf4f0f88da26050a95ade12a5"
server_key_version: "encryptionkey@0"
I20260924 12:47:23.205549 12185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:23.211730 12185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:23.212702 12185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:23.213078 12185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:23.230669 12185 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:44251
I20260924 12:47:23.230686 12250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:44251 every 8 connection(s)
I20260924 12:47:23.232064 12185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:47:23.235889 12251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:23.239950 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12185
I20260924 12:47:23.240145 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:47:23.240758 2373 external_mini_cluster.cc:1428] Setting key cb08dfe0596661b21ef03986ca17a2e2
I20260924 12:47:23.248023 12251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef: Bootstrap starting.
I20260924 12:47:23.250560 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:23.242853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.2.81.126:44251)
0924 12:47:23.243985 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:23.244001 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:23.244054 (+ 53us) server_negotiation.cc:407] Connection header received
0924 12:47:23.246440 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:23.246472 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:23.247015 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:23.247462 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:23.248449 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:23.248458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:23.248474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:23.248491 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:23.248575 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:23.248595 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:23.248797 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:23.248914 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 12:47:23.249221 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":345,"threads_started":1}
I20260924 12:47:23.251912 12251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:23.252964 12251 log.cc:824] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:23.256142 12251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef: No bootstrap required, opened a new log
I20260924 12:47:23.263720 12251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3254bdf486cc4cf0acec6de5561e04ef" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44251 } }
I20260924 12:47:23.264099 12251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:23.264191 12251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254bdf486cc4cf0acec6de5561e04ef, State: Initialized, Role: FOLLOWER
I20260924 12:47:23.264865 12251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3254bdf486cc4cf0acec6de5561e04ef" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44251 } }
I20260924 12:47:23.265192 12251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:47:23.265324 12251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:47:23.265473 12251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:23.267619 12251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3254bdf486cc4cf0acec6de5561e04ef" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44251 } }
I20260924 12:47:23.268143 12251 leader_election.cc:302] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3254bdf486cc4cf0acec6de5561e04ef; no voters:
I20260924 12:47:23.268743 12251 leader_election.cc:288] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:47:23.268874 12256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:23.269599 12256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [term 1 LEADER]: Becoming Leader. State: Replica: 3254bdf486cc4cf0acec6de5561e04ef, State: Running, Role: LEADER
I20260924 12:47:23.270237 12256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3254bdf486cc4cf0acec6de5561e04ef" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44251 } }
I20260924 12:47:23.270962 12251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:47:23.272997 12258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3254bdf486cc4cf0acec6de5561e04ef. Latest consensus state: current_term: 1 leader_uuid: "3254bdf486cc4cf0acec6de5561e04ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3254bdf486cc4cf0acec6de5561e04ef" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44251 } } }
I20260924 12:47:23.273093 12257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3254bdf486cc4cf0acec6de5561e04ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3254bdf486cc4cf0acec6de5561e04ef" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44251 } } }
I20260924 12:47:23.273830 12258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:23.273901 12257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:23.274744 12263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:47:23.280087 12263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:47:23.289988 12263 catalog_manager.cc:1347] Generated new cluster ID: f771c91136ff46728c96797c442070b5
I20260924 12:47:23.290124 12263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:47:23.304790 12263 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:47:23.305482 12263 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:47:23.321202 12263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3254bdf486cc4cf0acec6de5561e04ef: Generated new TSK 0
I20260924 12:47:23.322167 12263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:47:23.337160 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44251
--builtin_ntp_servers=127.2.81.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 12:47:23.662436 12275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:23.662734 12275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:23.662849 12275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:23.670243 12275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:23.670357 12275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:23.670496 12275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:47:23.680153 12275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:23.681483 12275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:23.683223 12275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:23.696292 12281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:23.696317 12283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:23.696261 12280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:23.697039 12275 server_base.cc:1034] running on GCE node
I20260924 12:47:23.697695 12275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:23.698693 12275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:23.699923 12275 hybrid_clock.cc:648] HybridClock initialized: now 1790254043699866 us; error 50 us; skew 500 ppm
I20260924 12:47:23.700292 12275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:23.702750 12275 webserver.cc:460] Webserver started at http://127.2.81.65:41161/ using document root <none> and password file <none>
I20260924 12:47:23.703449 12275 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:23.703563 12275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:23.703872 12275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:23.708261 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "ff7d670c0fa44f6a921489812c1c1486"
format_stamp: "Formatted at 2026-09-24 12:47:23 on dist-test-slave-9tm1"
server_key: "cc05e6f2f72197feb8760ed932b5b2b5"
server_key_iv: "2b6f100900d6acfb40d013531a922706"
server_key_version: "encryptionkey@0"
I20260924 12:47:23.709095 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "ff7d670c0fa44f6a921489812c1c1486"
format_stamp: "Formatted at 2026-09-24 12:47:23 on dist-test-slave-9tm1"
server_key: "cc05e6f2f72197feb8760ed932b5b2b5"
server_key_iv: "2b6f100900d6acfb40d013531a922706"
server_key_version: "encryptionkey@0"
I20260924 12:47:23.714243 12275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:47:23.717880 12289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:23.719426 12275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:47:23.719626 12275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "ff7d670c0fa44f6a921489812c1c1486"
format_stamp: "Formatted at 2026-09-24 12:47:23 on dist-test-slave-9tm1"
server_key: "cc05e6f2f72197feb8760ed932b5b2b5"
server_key_iv: "2b6f100900d6acfb40d013531a922706"
server_key_version: "encryptionkey@0"
I20260924 12:47:23.719848 12275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:23.726125 12275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:23.726959 12275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:23.727224 12275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:23.728119 12275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:23.729598 12275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:23.729692 12275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:23.729802 12275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:23.729871 12275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:23.761196 12275 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:44639
I20260924 12:47:23.761261 12401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:44639 every 8 connection(s)
I20260924 12:47:23.762530 12275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:47:23.769419 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12275
I20260924 12:47:23.769616 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:47:23.770277 2373 external_mini_cluster.cc:1428] Setting key e62fccd8dd0bbdd4925c24f3189f989f
I20260924 12:47:23.771353 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:23.765062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:55597 (local address 127.2.81.126:44251)
0924 12:47:23.765522 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:23.765526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:23.766828 (+ 1302us) server_negotiation.cc:407] Connection header received
0924 12:47:23.768679 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:23.768685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:23.768759 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:23.768858 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:23.770460 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:23.770468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:23.770472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:23.770494 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:23.770546 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:23.770593 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:23.770748 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:23.770986 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 12:47:23.771064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260924 12:47:23.772619 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:23.765694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44251 (local address 127.2.81.65:55597)
0924 12:47:23.766646 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:23.766692 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:23.768355 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:23.769220 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:23.769229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:23.769345 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:23.770062 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:47:23.770064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:23.770112 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:47:23.770245 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:23.770732 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:23.770742 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:23.770751 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:47:23.771044 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 12:47:23.771412 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":267,"threads_started":1}
I20260924 12:47:23.773247 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44251
--builtin_ntp_servers=127.2.81.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 12:47:23.774449 12402 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44251
I20260924 12:47:23.774771 12402 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:23.775377 12402 heartbeater.cc:511] Master 127.2.81.126:44251 requested a full tablet report, sending...
I20260924 12:47:23.777487 12216 ts_manager.cc:194] Registered new tserver with Master: ff7d670c0fa44f6a921489812c1c1486 (127.2.81.65:44639)
W20260924 12:47:24.093963 12406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:24.094267 12406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:24.094376 12406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:24.101845 12406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:24.101964 12406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:24.102103 12406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:47:24.111972 12406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:24.113329 12406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:24.115075 12406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:24.128103 12411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:24.128158 12412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:24.128137 12414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:24.128958 12406 server_base.cc:1034] running on GCE node
I20260924 12:47:24.129595 12406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:24.130527 12406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:24.131762 12406 hybrid_clock.cc:648] HybridClock initialized: now 1790254044131717 us; error 40 us; skew 500 ppm
I20260924 12:47:24.132107 12406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:24.134516 12406 webserver.cc:460] Webserver started at http://127.2.81.66:37429/ using document root <none> and password file <none>
I20260924 12:47:24.135210 12406 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:24.135301 12406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:24.135627 12406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:24.138504 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "76821082f5a541de87c3c567b589b11c"
format_stamp: "Formatted at 2026-09-24 12:47:24 on dist-test-slave-9tm1"
server_key: "8d39228392a1fe4a9029ccee53065af8"
server_key_iv: "f22a7b2fb6328e59615feeefa2412499"
server_key_version: "encryptionkey@0"
I20260924 12:47:24.139300 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "76821082f5a541de87c3c567b589b11c"
format_stamp: "Formatted at 2026-09-24 12:47:24 on dist-test-slave-9tm1"
server_key: "8d39228392a1fe4a9029ccee53065af8"
server_key_iv: "f22a7b2fb6328e59615feeefa2412499"
server_key_version: "encryptionkey@0"
I20260924 12:47:24.144569 12406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:47:24.148363 12420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:24.149911 12406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:47:24.150084 12406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "76821082f5a541de87c3c567b589b11c"
format_stamp: "Formatted at 2026-09-24 12:47:24 on dist-test-slave-9tm1"
server_key: "8d39228392a1fe4a9029ccee53065af8"
server_key_iv: "f22a7b2fb6328e59615feeefa2412499"
server_key_version: "encryptionkey@0"
I20260924 12:47:24.150308 12406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:24.155929 12406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:24.156728 12406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:24.157004 12406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:24.157940 12406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:24.159345 12406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:24.159440 12406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:24.159547 12406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:24.159621 12406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:24.189576 12406 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:37705
I20260924 12:47:24.189648 12532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:37705 every 8 connection(s)
I20260924 12:47:24.190886 12406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:47:24.195147 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12406
I20260924 12:47:24.195348 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:47:24.195937 2373 external_mini_cluster.cc:1428] Setting key a71308a9b88bd460ba03e6c4792c70d2
I20260924 12:47:24.199256 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44251
--builtin_ntp_servers=127.2.81.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 12:47:24.200359 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.193625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:40235 (local address 127.2.81.126:44251)
0924 12:47:24.193828 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:24.193832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:24.195458 (+ 1626us) server_negotiation.cc:407] Connection header received
0924 12:47:24.197639 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:24.197645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:24.197727 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:24.197838 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:24.199533 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:24.199538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:24.199541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:24.199562 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:24.199605 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:24.199617 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:24.199725 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:24.200025 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 12:47:24.200129 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 12:47:24.202191 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.194147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44251 (local address 127.2.81.66:40235)
0924 12:47:24.195276 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:24.195323 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:24.197298 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:24.198127 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:24.198140 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:24.198338 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:24.199087 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:47:24.199091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:24.199150 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:47:24.199297 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:24.199752 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:24.199764 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:24.199773 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:47:24.200094 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 12:47:24.200484 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":289,"threads_started":1}
I20260924 12:47:24.203967 12533 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44251
I20260924 12:47:24.204326 12533 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:24.205214 12533 heartbeater.cc:511] Master 127.2.81.126:44251 requested a full tablet report, sending...
I20260924 12:47:24.207302 12216 ts_manager.cc:194] Registered new tserver with Master: 76821082f5a541de87c3c567b589b11c (127.2.81.66:37705)
W20260924 12:47:24.554780 12536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:24.555145 12536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:24.555294 12536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:24.564276 12536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:24.564448 12536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:24.564637 12536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:47:24.575259 12536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:44251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:24.577291 12536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:24.579316 12536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:24.598057 12541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:24.598057 12542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:24.598758 12544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:24.599859 12536 server_base.cc:1034] running on GCE node
I20260924 12:47:24.600649 12536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:24.601868 12536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:24.603112 12536 hybrid_clock.cc:648] HybridClock initialized: now 1790254044603046 us; error 58 us; skew 500 ppm
I20260924 12:47:24.603767 12536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:47:24.606549 12536 webserver.cc:460] Webserver started at http://127.2.81.67:36055/ using document root <none> and password file <none>
I20260924 12:47:24.607326 12536 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:24.607429 12536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:24.607717 12536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:24.611161 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "9e7c2ed3241e4879986a6bb5f07252dd"
format_stamp: "Formatted at 2026-09-24 12:47:24 on dist-test-slave-9tm1"
server_key: "5269cc4958ffe063a480f637fd63f4bd"
server_key_iv: "db83fb6ec6ca441a7c149311d5ebe7bc"
server_key_version: "encryptionkey@0"
I20260924 12:47:24.612035 12536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "9e7c2ed3241e4879986a6bb5f07252dd"
format_stamp: "Formatted at 2026-09-24 12:47:24 on dist-test-slave-9tm1"
server_key: "5269cc4958ffe063a480f637fd63f4bd"
server_key_iv: "db83fb6ec6ca441a7c149311d5ebe7bc"
server_key_version: "encryptionkey@0"
I20260924 12:47:24.619453 12536 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260924 12:47:24.624050 12550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:24.626451 12536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 12:47:24.626783 12536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "9e7c2ed3241e4879986a6bb5f07252dd"
format_stamp: "Formatted at 2026-09-24 12:47:24 on dist-test-slave-9tm1"
server_key: "5269cc4958ffe063a480f637fd63f4bd"
server_key_iv: "db83fb6ec6ca441a7c149311d5ebe7bc"
server_key_version: "encryptionkey@0"
I20260924 12:47:24.627015 12536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:24.633487 12536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:24.634426 12536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:24.634677 12536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:24.636430 12536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:24.638511 12536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:24.638633 12536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:24.638731 12536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:24.638787 12536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:24.677753 12536 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:41145
I20260924 12:47:24.677801 12662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:41145 every 8 connection(s)
I20260924 12:47:24.679090 12536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:47:24.686738 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12536
I20260924 12:47:24.686945 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:47:24.687163 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.681636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:40431 (local address 127.2.81.126:44251)
0924 12:47:24.681818 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:24.681823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:24.683204 (+ 1381us) server_negotiation.cc:407] Connection header received
0924 12:47:24.684822 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:24.684826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:24.684878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:24.684967 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:24.686345 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:24.686352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:24.686357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:24.686378 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:24.686419 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:24.686436 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:24.686584 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:24.686785 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 12:47:24.686854 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:47:24.687484 2373 external_mini_cluster.cc:1428] Setting key 7843e66372d5ca498eaadc1dd749de97
I20260924 12:47:24.688185 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.682161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:44251 (local address 127.2.81.67:40431)
0924 12:47:24.683050 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:24.683087 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:24.684553 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:24.685216 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:24.685224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:24.685327 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:24.685987 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:47:24.685989 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:24.686027 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:47:24.686113 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:24.686550 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:24.686558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:24.686565 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:47:24.686820 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 12:47:24.687092 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":250,"threads_started":1}
I20260924 12:47:24.689646 12663 heartbeater.cc:347] Connected to a master server at 127.2.81.126:44251
I20260924 12:47:24.689942 12663 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:24.690541 12663 heartbeater.cc:511] Master 127.2.81.126:44251 requested a full tablet report, sending...
I20260924 12:47:24.691898 12216 ts_manager.cc:194] Registered new tserver with Master: 9e7c2ed3241e4879986a6bb5f07252dd (127.2.81.67:41145)
I20260924 12:47:24.702894 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 12:47:24.708649 12404 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:47:24.707214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.2.81.126:44251)
0924 12:47:24.707366 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:24.707372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:24.707824 (+ 452us) server_negotiation.cc:407] Connection header received
0924 12:47:24.707994 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:24.707998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:24.708051 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:24.708133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:24.708444 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48632: BlockingRecv error: recv got EOF from 127.0.0.1:48632 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 12:47:24.709596 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12275
I20260924 12:47:24.712785 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.710532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.2.81.65:44639)
0924 12:47:24.711004 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:24.711011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:24.711036 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:47:24.711093 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:24.711100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:24.711320 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:24.711445 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:24.712172 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:24.712178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:24.712189 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:24.712204 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:24.712239 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:24.712252 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:24.712353 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:24.712404 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 12:47:24.712483 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260924 12:47:24.714140 12397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:24.793164 12402 heartbeater.cc:503] Master 127.2.81.126:44251 was elected leader, sending a full tablet report...
I20260924 12:47:24.794791 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12275
I20260924 12:47:24.807551 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12406
I20260924 12:47:24.810962 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.808387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.2.81.66:37705)
0924 12:47:24.808860 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:24.808868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:24.808893 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:47:24.808952 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:24.808958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:24.809233 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:24.809366 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:24.810316 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:24.810323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:24.810335 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:24.810348 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:24.810392 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:24.810405 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:24.810514 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:24.810654 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 12:47:24.810741 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":211,"threads_started":1}
I20260924 12:47:24.812161 12528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:24.893153 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12406
I20260924 12:47:24.906669 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12536
I20260924 12:47:24.910372 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:24.907559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.2.81.67:41145)
0924 12:47:24.908177 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:24.908184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:24.908213 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:47:24.908271 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:24.908277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:24.908490 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:24.908624 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:24.909670 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:24.909676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:24.909687 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:47:24.909703 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:24.909738 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:47:24.909754 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:24.909906 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:24.909957 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 12:47:24.910040 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":472,"threads_started":1}
I20260924 12:47:24.911587 12658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:24.992354 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12536
I20260924 12:47:25.004426 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12185
I20260924 12:47:25.005656 12246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:25.106648 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12185
2026-09-24T12:47:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2354 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:47:25 dist-test-slave-9tm1 krb5kdc[12681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:47:25 dist-test-slave-9tm1 krb5kdc[12681](info): set up 2 sockets
Sep 24 12:47:25 dist-test-slave-9tm1 krb5kdc[12681](info): commencing operation
krb5kdc: starting...
W20260924 12:47:26.149947 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:47:26 dist-test-slave-9tm1 krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:47:26.389624 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33703
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:26.718194 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:26.718530 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:26.718662 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:26.726147 12694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:47:26.726296 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:26.726375 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:26.726451 12694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:47:26.726512 12694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:47:26.736526 12694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:33703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:26.737900 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:26.739238 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:26.750852 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:26.750888 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:26.750815 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:26.751915 12694 server_base.cc:1034] running on GCE node
I20260924 12:47:26.752699 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:26.754158 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:26.755358 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790254046755304 us; error 49 us; skew 500 ppm
I20260924 12:47:26.755677 12694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:26 dist-test-slave-9tm1 krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:26.758553 12694 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:47:26.760003 12694 webserver.cc:460] Webserver started at http://127.2.81.126:32833/ using document root <none> and password file <none>
I20260924 12:47:26.760634 12694 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:26.760730 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:26.761003 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:26.763849 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "d5af0308b644479890c19c8b0b513f5a"
format_stamp: "Formatted at 2026-09-24 12:47:26 on dist-test-slave-9tm1"
server_key: "5a1fba4a9e8193b0cc55a08e7eb54363"
server_key_iv: "10784162236f5d554880539069726dca"
server_key_version: "encryptionkey@0"
I20260924 12:47:26.764557 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "d5af0308b644479890c19c8b0b513f5a"
format_stamp: "Formatted at 2026-09-24 12:47:26 on dist-test-slave-9tm1"
server_key: "5a1fba4a9e8193b0cc55a08e7eb54363"
server_key_iv: "10784162236f5d554880539069726dca"
server_key_version: "encryptionkey@0"
I20260924 12:47:26.769722 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 12:47:26.773102 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:26.774519 12694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:47:26.774688 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "d5af0308b644479890c19c8b0b513f5a"
format_stamp: "Formatted at 2026-09-24 12:47:26 on dist-test-slave-9tm1"
server_key: "5a1fba4a9e8193b0cc55a08e7eb54363"
server_key_iv: "10784162236f5d554880539069726dca"
server_key_version: "encryptionkey@0"
I20260924 12:47:26.774884 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:26.799396 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:26.803750 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:26.804021 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:26.821756 12694 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:33703
I20260924 12:47:26.821779 12760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:33703 every 8 connection(s)
I20260924 12:47:26.823041 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:47:26.826630 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:26.831786 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12694
I20260924 12:47:26.832015 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:47:26.832715 2373 external_mini_cluster.cc:1428] Setting key 70359060b4abb99ae67f8aa4549f6949
I20260924 12:47:26.837934 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a: Bootstrap starting.
I20260924 12:47:26.841449 12761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:26.842691 12761 log.cc:824] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:26.846024 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a: No bootstrap required, opened a new log
I20260924 12:47:26.854321 12761 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af0308b644479890c19c8b0b513f5a" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33703 } }
Sep 24 12:47:26 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:26.854748 12761 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:26.854878 12761 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5af0308b644479890c19c8b0b513f5a, State: Initialized, Role: FOLLOWER
I20260924 12:47:26.855693 12761 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af0308b644479890c19c8b0b513f5a" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33703 } }
I20260924 12:47:26.855991 12761 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:47:26.856140 12761 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:47:26.856318 12761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:26.859087 12761 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af0308b644479890c19c8b0b513f5a" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33703 } }
I20260924 12:47:26.859826 12761 leader_election.cc:302] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5af0308b644479890c19c8b0b513f5a; no voters:
I20260924 12:47:26.860638 12761 leader_election.cc:288] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:47:26.860756 12766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:26.861832 12766 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [term 1 LEADER]: Becoming Leader. State: Replica: d5af0308b644479890c19c8b0b513f5a, State: Running, Role: LEADER
I20260924 12:47:26.862505 12766 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af0308b644479890c19c8b0b513f5a" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33703 } }
I20260924 12:47:26.862988 12761 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:47:26.865494 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:26.834845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.2.81.126:33703)
0924 12:47:26.835644 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:26.835659 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:26.835712 (+ 53us) server_negotiation.cc:407] Connection header received
0924 12:47:26.837403 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:26.837427 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:26.837849 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:26.838231 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:26.839905 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:26.841355 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:26.843118 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:26.844164 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:26.844331 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:26.856143 (+ 11812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:26.856155 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:26.856174 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:26.856240 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:26.859480 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:26.860640 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:26.860652 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:26.860662 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:26.860929 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:26.861487 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:26.861493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:26.861497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:26.862315 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:26.862667 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:26.863187 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 12:47:26.863657 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":389,"threads_started":1}
I20260924 12:47:26.866605 12768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5af0308b644479890c19c8b0b513f5a. Latest consensus state: current_term: 1 leader_uuid: "d5af0308b644479890c19c8b0b513f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af0308b644479890c19c8b0b513f5a" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33703 } } }
I20260924 12:47:26.866895 12768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:26.867403 12767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5af0308b644479890c19c8b0b513f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af0308b644479890c19c8b0b513f5a" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33703 } } }
I20260924 12:47:26.867734 12767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a [sys.catalog]: This master's current role is: LEADER
I20260924 12:47:26.867834 12771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:47:26.885267 12771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:47:26.894362 12771 catalog_manager.cc:1347] Generated new cluster ID: 9949454d5dd94793840c41177729977e
I20260924 12:47:26.894497 12771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:47:26.914745 12771 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:47:26.916093 12771 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:47:26.930881 12771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a: Generated new TSK 0
I20260924 12:47:26.932052 12771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:47:26.989946 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33703
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:27.327364 12787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:27.327679 12787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:27.327817 12787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:27.335559 12787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:27.335738 12787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:27.335897 12787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:47:27.346159 12787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:27.347532 12787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:27.348845 12787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:27.362255 12792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:27.362241 12795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:27.362241 12793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:27.362978 12787 server_base.cc:1034] running on GCE node
I20260924 12:47:27.363638 12787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:27.364701 12787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:27.365898 12787 hybrid_clock.cc:648] HybridClock initialized: now 1790254047365837 us; error 57 us; skew 500 ppm
I20260924 12:47:27.366230 12787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:27 dist-test-slave-9tm1 krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:27.369364 12787 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:47:27.370807 12787 webserver.cc:460] Webserver started at http://127.2.81.65:41145/ using document root <none> and password file <none>
I20260924 12:47:27.371456 12787 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:27.371551 12787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:27.371791 12787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:27.374644 12787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "c28c72131a494f0b880d82f65f52cd70"
format_stamp: "Formatted at 2026-09-24 12:47:27 on dist-test-slave-9tm1"
server_key: "4d376f3e3d4b9b9605ec4cebb78d5fc5"
server_key_iv: "c5225d0a966478d9a2ea157a40775bf5"
server_key_version: "encryptionkey@0"
I20260924 12:47:27.375345 12787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "c28c72131a494f0b880d82f65f52cd70"
format_stamp: "Formatted at 2026-09-24 12:47:27 on dist-test-slave-9tm1"
server_key: "4d376f3e3d4b9b9605ec4cebb78d5fc5"
server_key_iv: "c5225d0a966478d9a2ea157a40775bf5"
server_key_version: "encryptionkey@0"
I20260924 12:47:27.380465 12787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 12:47:27.383985 12802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:27.385387 12787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:47:27.385514 12787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "c28c72131a494f0b880d82f65f52cd70"
format_stamp: "Formatted at 2026-09-24 12:47:27 on dist-test-slave-9tm1"
server_key: "4d376f3e3d4b9b9605ec4cebb78d5fc5"
server_key_iv: "c5225d0a966478d9a2ea157a40775bf5"
server_key_version: "encryptionkey@0"
I20260924 12:47:27.385679 12787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:27.400281 12787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:27.404484 12787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:27.404750 12787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:27.405735 12787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:27.407035 12787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:27.407104 12787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:27.407204 12787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:27.407271 12787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:27.437793 12787 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:36745
I20260924 12:47:27.437831 12914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:36745 every 8 connection(s)
I20260924 12:47:27.439034 12787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:47:27.445384 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12787
I20260924 12:47:27.445567 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:47:27.446148 2373 external_mini_cluster.cc:1428] Setting key 671d45141761b1bc2fc666c19da775ef
Sep 24 12:47:27 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:27.460479 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:27.441656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:42789 (local address 127.2.81.126:33703)
0924 12:47:27.442087 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:27.442092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:27.442957 (+ 865us) server_negotiation.cc:407] Connection header received
0924 12:47:27.444682 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:27.444687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:27.444776 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:27.444868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:27.446474 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.447360 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.449624 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.450036 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.450146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:27.454277 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:27.454284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:27.454287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:27.454323 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:27.457231 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:27.458009 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:27.458013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:27.458017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:27.458167 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:27.458567 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:27.458571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:27.458575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:27.458884 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:27.459014 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:27.459846 (+ 832us) server_negotiation.cc:299] Negotiation successful
0924 12:47:27.460079 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260924 12:47:27.461169 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:27.442097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33703 (local address 127.2.81.65:42789)
0924 12:47:27.442799 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:27.442841 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:27.444404 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:27.445129 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:27.445146 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:27.445710 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:27.446309 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:27.446328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.447532 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.447536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:27.449428 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:27.449446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.450273 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.450279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:27.450561 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:27.451304 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:27.451331 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:27.453978 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:27.457402 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:27.457411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:27.457428 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:27.457876 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:27.458262 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:27.458266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:27.458268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:27.458461 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:27.458986 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:27.458992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:27.459334 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 12:47:27.459568 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 12:47:27.459979 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":270,"threads_started":1}
I20260924 12:47:27.462812 12915 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33703
I20260924 12:47:27.463166 12915 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:27.464111 12915 heartbeater.cc:511] Master 127.2.81.126:33703 requested a full tablet report, sending...
I20260924 12:47:27.466554 12726 ts_manager.cc:194] Registered new tserver with Master: c28c72131a494f0b880d82f65f52cd70 (127.2.81.65:36745)
I20260924 12:47:27.467998 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:42789
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:47:27.500854 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33703
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:27.830766 12921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:27.831121 12921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:27.831264 12921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:27.838841 12921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:27.838987 12921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:27.839148 12921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:47:27.850579 12921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:27.852131 12921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:27.853665 12921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:27.867112 12926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:27.867132 12929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:27.867069 12927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:27.867755 12921 server_base.cc:1034] running on GCE node
I20260924 12:47:27.868464 12921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:27.869529 12921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:27.870795 12921 hybrid_clock.cc:648] HybridClock initialized: now 1790254047870729 us; error 50 us; skew 500 ppm
I20260924 12:47:27.871204 12921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:27 dist-test-slave-9tm1 krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:27.874871 12921 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:47:27.876478 12921 webserver.cc:460] Webserver started at http://127.2.81.66:33905/ using document root <none> and password file <none>
I20260924 12:47:27.877283 12921 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:27.877413 12921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:27.877700 12921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:27.880806 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "3f832f99a8584630bc5110abbe4366dc"
format_stamp: "Formatted at 2026-09-24 12:47:27 on dist-test-slave-9tm1"
server_key: "8e173d1b6801d3c4bce996c7d77a6021"
server_key_iv: "797aed94c8d23567af6d42b450428b3b"
server_key_version: "encryptionkey@0"
I20260924 12:47:27.881636 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "3f832f99a8584630bc5110abbe4366dc"
format_stamp: "Formatted at 2026-09-24 12:47:27 on dist-test-slave-9tm1"
server_key: "8e173d1b6801d3c4bce996c7d77a6021"
server_key_iv: "797aed94c8d23567af6d42b450428b3b"
server_key_version: "encryptionkey@0"
I20260924 12:47:27.887068 12921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:47:27.891559 12936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:27.893261 12921 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:47:27.893452 12921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "3f832f99a8584630bc5110abbe4366dc"
format_stamp: "Formatted at 2026-09-24 12:47:27 on dist-test-slave-9tm1"
server_key: "8e173d1b6801d3c4bce996c7d77a6021"
server_key_iv: "797aed94c8d23567af6d42b450428b3b"
server_key_version: "encryptionkey@0"
I20260924 12:47:27.893667 12921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:27.919840 12921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:27.924250 12921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:27.924525 12921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:27.925554 12921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:27.926970 12921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:27.927052 12921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:27.927182 12921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:27.927237 12921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:27.961472 12921 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:41011
I20260924 12:47:27.961529 13048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:41011 every 8 connection(s)
I20260924 12:47:27.962747 12921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:47:27.963408 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 12921
I20260924 12:47:27.963618 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:47:27.964228 2373 external_mini_cluster.cc:1428] Setting key a43d1731422bf9ee96c3bcedfd504a0b
Sep 24 12:47:27 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:27.987807 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:27.965729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:34899 (local address 127.2.81.126:33703)
0924 12:47:27.966176 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:27.966181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:27.969850 (+ 3669us) server_negotiation.cc:407] Connection header received
0924 12:47:27.971861 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:27.971867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:27.971973 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:27.972097 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:27.974068 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.974821 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.976508 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.976949 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.977121 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:27.981309 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:27.981315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:27.981319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:27.981357 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:27.984013 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:27.985159 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:27.985164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:27.985166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:27.985327 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:27.985876 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:27.985879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:27.985881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:27.986196 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:27.986312 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:27.987331 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0924 12:47:27.987527 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":246,"threads_started":1}
I20260924 12:47:27.989516 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:27.966435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33703 (local address 127.2.81.66:34899)
0924 12:47:27.969653 (+ 3218us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:27.969703 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:27.971492 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:27.972393 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:27.972404 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:27.973115 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:27.973847 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:27.973873 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.974991 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.974995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:27.976325 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:27.976342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:27.977231 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:27.977238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:27.977490 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:27.978217 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:27.978244 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:27.980904 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:27.984217 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:27.984228 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:27.984248 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:27.984917 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:27.985458 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:27.985463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:27.985467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:27.985749 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:27.986338 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:27.986347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:27.986848 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 12:47:27.987202 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 12:47:27.987800 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2911,"thread_start_us":2699,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:47:27.991483 13049 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33703
I20260924 12:47:27.991819 13049 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:27.992749 13049 heartbeater.cc:511] Master 127.2.81.126:33703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:47:27.994676 12726 ts_manager.cc:194] Registered new tserver with Master: 3f832f99a8584630bc5110abbe4366dc (127.2.81.66:41011)
I20260924 12:47:27.995551 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:34899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:47:28.026326 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33703
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:47:28.371768 13055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:47:28.372097 13055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:47:28.372227 13055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:47:28.379846 13055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:47:28.379997 13055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:47:28.380112 13055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:47:28.390611 13055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:47:28.391961 13055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:47:28.393388 13055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:47:28.406689 13060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:28.406749 13061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:47:28.406728 13063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:47:28.407852 13055 server_base.cc:1034] running on GCE node
I20260924 12:47:28.408668 13055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:47:28.410347 13055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:47:28.411553 13055 hybrid_clock.cc:648] HybridClock initialized: now 1790254048411491 us; error 65 us; skew 500 ppm
I20260924 12:47:28.411866 13055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:47:28 dist-test-slave-9tm1 krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254048, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:47:28.415105 13055 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:47:28.416649 13055 webserver.cc:460] Webserver started at http://127.2.81.67:46477/ using document root <none> and password file <none>
I20260924 12:47:28.417385 13055 fs_manager.cc:359] Metadata directory not provided
I20260924 12:47:28.417507 13055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:47:28.417802 13055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:47:28.420805 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "84f9e40dcb8a48e5942f78e359229fb3"
format_stamp: "Formatted at 2026-09-24 12:47:28 on dist-test-slave-9tm1"
server_key: "76736a2f2e88d74d32b27d6e13c554c4"
server_key_iv: "f3c12706739d537812f40b038cabe3ab"
server_key_version: "encryptionkey@0"
I20260924 12:47:28.421639 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "84f9e40dcb8a48e5942f78e359229fb3"
format_stamp: "Formatted at 2026-09-24 12:47:28 on dist-test-slave-9tm1"
server_key: "76736a2f2e88d74d32b27d6e13c554c4"
server_key_iv: "f3c12706739d537812f40b038cabe3ab"
server_key_version: "encryptionkey@0"
I20260924 12:47:28.426874 13055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:47:28.430451 13070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:28.431939 13055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 12:47:28.432122 13055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "84f9e40dcb8a48e5942f78e359229fb3"
format_stamp: "Formatted at 2026-09-24 12:47:28 on dist-test-slave-9tm1"
server_key: "76736a2f2e88d74d32b27d6e13c554c4"
server_key_iv: "f3c12706739d537812f40b038cabe3ab"
server_key_version: "encryptionkey@0"
I20260924 12:47:28.432336 13055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:47:28.452816 13055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:47:28.457506 13055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:47:28.457787 13055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:47:28.458832 13055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:47:28.460282 13055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:47:28.460361 13055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:28.460484 13055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:47:28.460536 13055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:47:28.471540 12915 heartbeater.cc:503] Master 127.2.81.126:33703 was elected leader, sending a full tablet report...
I20260924 12:47:28.492123 13055 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:42817
I20260924 12:47:28.492218 13182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:42817 every 8 connection(s)
I20260924 12:47:28.493492 13055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:47:28.500435 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 13055
I20260924 12:47:28.500653 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:47:28.501086 2373 external_mini_cluster.cc:1428] Setting key 5c59400504a2fd671898574439ef7eee
Sep 24 12:47:28 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254048, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:47:28.514362 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.496010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:51899 (local address 127.2.81.126:33703)
0924 12:47:28.496497 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.496502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.497719 (+ 1217us) server_negotiation.cc:407] Connection header received
0924 12:47:28.499733 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.499738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.499833 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.499937 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:28.501742 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.502642 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.504617 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.505073 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.505194 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.508948 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:28.508954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:28.508957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:28.509044 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:28.511612 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.512376 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.512380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.512382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.512491 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.512830 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.512833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.512835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.513129 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:28.513251 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.513890 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.514055 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
I20260924 12:47:28.515679 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.496587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:33703 (local address 127.2.81.67:51899)
0924 12:47:28.497547 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:28.497587 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:28.499413 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:28.500152 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:28.500161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:28.500809 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:47:28.501515 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.501537 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.502805 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.502809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.504421 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.504435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.505377 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.505383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.505641 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.506345 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:47:28.506378 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:47:28.508696 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:47:28.511766 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:28.511774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:28.511789 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:28.512276 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:28.512568 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:47:28.512571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:47:28.512573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:47:28.512752 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:47:28.513257 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:47:28.513264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:47:28.513593 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 12:47:28.513828 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 12:47:28.514239 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":446,"threads_started":1}
I20260924 12:47:28.517270 13183 heartbeater.cc:347] Connected to a master server at 127.2.81.126:33703
I20260924 12:47:28.517560 13183 heartbeater.cc:464] Registering TS with master...
I20260924 12:47:28.518301 13183 heartbeater.cc:511] Master 127.2.81.126:33703 requested a full tablet report, sending...
I20260924 12:47:28.519869 12726 ts_manager.cc:194] Registered new tserver with Master: 84f9e40dcb8a48e5942f78e359229fb3 (127.2.81.67:42817)
I20260924 12:47:28.520673 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:51899
I20260924 12:47:28.529313 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:47:28.546470 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.533813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.2.81.126:33703)
0924 12:47:28.534051 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.534059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.534417 (+ 358us) server_negotiation.cc:407] Connection header received
0924 12:47:28.534587 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.534591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.534659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.534744 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:28.536190 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.537011 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.538261 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.538656 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.538772 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.541060 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:28.541066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:28.541069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:28.541107 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:28.543826 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.544514 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.544519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.544522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.544659 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.545042 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.545047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.545050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.545375 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:28.545611 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.545954 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.546172 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:47:28.551226 12726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:47:28.558554 12726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:47:28.584227 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.571233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.2.81.65:36745)
0924 12:47:28.573481 (+ 2248us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.573490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.573516 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:47:28.573622 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.573675 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.573931 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.575930 (+ 1999us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:47:28.576190 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.579005 (+ 2815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.581490 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.583028 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.583174 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.583407 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.583706 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.583885 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2095,"thread_start_us":1897,"threads_started":1}
I20260924 12:47:28.584311 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.571636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.2.81.66:41011)
0924 12:47:28.574294 (+ 2658us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.574301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.574790 (+ 489us) server_negotiation.cc:407] Connection header received
0924 12:47:28.575074 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.575081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.575332 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.575498 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:47:28.576364 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.577820 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.581634 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.583024 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.583167 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.583374 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.583841 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.583994 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2518,"thread_start_us":3074,"threads_started":1}
I20260924 12:47:28.584302 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.570898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36745 (local address 127.0.0.1:48586)
0924 12:47:28.572634 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:28.572678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:28.572866 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:28.575211 (+ 2345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:28.575218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:28.575245 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:47:28.575718 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.575731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.579157 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.579163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.581332 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.581343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.583244 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.583251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.583458 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.583512 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 12:47:28.583643 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 12:47:28.583838 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"mutex_wait_us":48,"thread_start_us":3511,"threads_started":1}
I20260924 12:47:28.585111 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.571127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:41011 (local address 127.0.0.1:41954)
0924 12:47:28.574687 (+ 3560us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:28.574704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:28.574885 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:28.575774 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:28.575778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:28.575800 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:47:28.576186 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.576198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.577979 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.577984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.581459 (+ 3475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.581473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.583525 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.583529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.583650 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.583679 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 12:47:28.584558 (+ 879us) client_negotiation.cc:240] Negotiation successful
0924 12:47:28.584702 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3450,"mutex_wait_us":26,"thread_start_us":1030,"threads_started":1}
I20260924 12:47:28.589309 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.574430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.2.81.67:42817)
0924 12:47:28.580292 (+ 5862us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.580302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.580868 (+ 566us) server_negotiation.cc:407] Connection header received
0924 12:47:28.581096 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.581113 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.581396 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.581609 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:47:28.582266 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.583758 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.586312 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.587707 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.587843 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.588089 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.588774 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.588941 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5672,"thread_start_us":5524,"threads_started":1}
I20260924 12:47:28.588893 12984 tablet_service.cc:1467] Processing CreateTablet for tablet 64baf1bc208d40349d75b234f3182f04 (DEFAULT_TABLE table=test-table [id=e427d2a61bac49379e5d890c095f5f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:47:28.588781 12850 tablet_service.cc:1467] Processing CreateTablet for tablet 64baf1bc208d40349d75b234f3182f04 (DEFAULT_TABLE table=test-table [id=e427d2a61bac49379e5d890c095f5f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:47:28.589924 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.574131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:42817 (local address 127.0.0.1:38144)
0924 12:47:28.580729 (+ 6598us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:28.580748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:28.580889 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:28.581646 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:28.581650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:28.581676 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:47:28.582072 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.582084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.583901 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.583906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.586131 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.586148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.588231 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.588238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.588416 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.588450 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:47:28.588544 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 12:47:28.589541 (+ 997us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6473,"thread_start_us":6308,"threads_started":1}
I20260924 12:47:28.590478 12984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64baf1bc208d40349d75b234f3182f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:28.590806 12850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64baf1bc208d40349d75b234f3182f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:28.594872 13118 tablet_service.cc:1467] Processing CreateTablet for tablet 64baf1bc208d40349d75b234f3182f04 (DEFAULT_TABLE table=test-table [id=e427d2a61bac49379e5d890c095f5f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:47:28.596572 13118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64baf1bc208d40349d75b234f3182f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:47:28.605631 13202 tablet_bootstrap.cc:492] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Bootstrap starting.
I20260924 12:47:28.608556 13203 tablet_bootstrap.cc:492] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Bootstrap starting.
I20260924 12:47:28.609717 13202 tablet_bootstrap.cc:654] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:28.611212 13202 log.cc:824] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:28.611589 13204 tablet_bootstrap.cc:492] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Bootstrap starting.
I20260924 12:47:28.612191 13203 tablet_bootstrap.cc:654] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:28.613559 13203 log.cc:824] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:28.615038 13204 tablet_bootstrap.cc:654] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Neither blocks nor log segments found. Creating new log.
I20260924 12:47:28.615175 13202 tablet_bootstrap.cc:492] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: No bootstrap required, opened a new log
I20260924 12:47:28.615423 13202 ts_tablet_manager.cc:1397] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 12:47:28.616235 13204 log.cc:824] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Log is configured to *not* fsync() on all Append() calls
I20260924 12:47:28.616602 13203 tablet_bootstrap.cc:492] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: No bootstrap required, opened a new log
I20260924 12:47:28.616885 13203 ts_tablet_manager.cc:1397] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:47:28.623200 13202 raft_consensus.cc:348] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.623592 13202 raft_consensus.cc:374] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:28.623682 13202 raft_consensus.cc:729] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f832f99a8584630bc5110abbe4366dc, State: Initialized, Role: FOLLOWER
I20260924 12:47:28.623840 13204 tablet_bootstrap.cc:492] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: No bootstrap required, opened a new log
I20260924 12:47:28.624153 13204 ts_tablet_manager.cc:1397] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260924 12:47:28.624377 13202 consensus_queue.cc:260] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.625440 13049 heartbeater.cc:503] Master 127.2.81.126:33703 was elected leader, sending a full tablet report...
I20260924 12:47:28.627619 13202 ts_tablet_manager.cc:1428] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 12:47:28.627327 13203 raft_consensus.cc:348] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.627769 13203 raft_consensus.cc:374] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:28.627854 13203 raft_consensus.cc:729] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28c72131a494f0b880d82f65f52cd70, State: Initialized, Role: FOLLOWER
I20260924 12:47:28.628289 13208 raft_consensus.cc:482] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:47:28.628599 13208 raft_consensus.cc:504] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.628618 13203 consensus_queue.cc:260] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.633330 13208 leader_election.cc:288] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c28c72131a494f0b880d82f65f52cd70 (127.2.81.65:36745), 84f9e40dcb8a48e5942f78e359229fb3 (127.2.81.67:42817)
I20260924 12:47:28.635972 13204 raft_consensus.cc:348] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.636381 13204 raft_consensus.cc:374] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:47:28.636483 13204 raft_consensus.cc:729] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f9e40dcb8a48e5942f78e359229fb3, State: Initialized, Role: FOLLOWER
I20260924 12:47:28.637122 13204 consensus_queue.cc:260] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.638157 13183 heartbeater.cc:503] Master 127.2.81.126:33703 was elected leader, sending a full tablet report...
I20260924 12:47:28.638681 13204 ts_tablet_manager.cc:1428] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 12:47:28.641755 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.631837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:43917 (local address 127.2.81.65:36745)
0924 12:47:28.632031 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.632055 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.634140 (+ 2085us) server_negotiation.cc:407] Connection header received
0924 12:47:28.634419 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.634425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.634500 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.634572 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:47:28.635318 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.636496 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.639155 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.640465 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.640650 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.640768 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.641418 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.641556 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 12:47:28.641913 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.631376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36745 (local address 127.2.81.66:43917)
0924 12:47:28.634038 (+ 2662us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:28.634059 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:28.634256 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:28.634767 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:28.634772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:28.634799 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:47:28.635148 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.635159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.636701 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.636708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.638925 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.638941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.640944 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.640952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.641213 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.641251 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:47:28.641361 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 12:47:28.641518 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2526,"thread_start_us":2352,"threads_started":1}
I20260924 12:47:28.642997 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64baf1bc208d40349d75b234f3182f04" candidate_uuid: "3f832f99a8584630bc5110abbe4366dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28c72131a494f0b880d82f65f52cd70" is_pre_election: true
I20260924 12:47:28.643682 12870 raft_consensus.cc:2454] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f832f99a8584630bc5110abbe4366dc in term 0.
I20260924 12:47:28.644516 12939 leader_election.cc:302] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f832f99a8584630bc5110abbe4366dc, c28c72131a494f0b880d82f65f52cd70; no voters:
I20260924 12:47:28.645063 13208 raft_consensus.cc:2781] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:47:28.645247 13208 raft_consensus.cc:482] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:47:28.645362 13208 raft_consensus.cc:3037] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:28.647213 13203 ts_tablet_manager.cc:1428] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Time spent starting tablet: real 0.030s user 0.015s sys 0.003s
I20260924 12:47:28.648873 13208 raft_consensus.cc:504] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.649116 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.634094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:38101 (local address 127.2.81.67:42817)
0924 12:47:28.634290 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.634296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.642249 (+ 7953us) server_negotiation.cc:407] Connection header received
0924 12:47:28.643027 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.643033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.643116 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.643177 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:47:28.643742 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.644701 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.647131 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.648165 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.648242 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.648314 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.648724 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.648850 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:47:28.649185 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.633616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:42817 (local address 127.2.81.66:38101)
0924 12:47:28.642163 (+ 8547us) negotiation.cc:107] Waiting for socket to connect
0924 12:47:28.642182 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:47:28.642844 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:47:28.643362 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:47:28.643368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:47:28.643395 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:47:28.643598 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.643609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.644860 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.644865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.646909 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:47:28.646942 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.648426 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.648431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:47:28.648573 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.648596 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 12:47:28.648683 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 12:47:28.648832 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8447,"thread_start_us":11636,"threads_started":1}
I20260924 12:47:28.650012 13208 leader_election.cc:288] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [CANDIDATE]: Term 1 election: Requested vote from peers c28c72131a494f0b880d82f65f52cd70 (127.2.81.65:36745), 84f9e40dcb8a48e5942f78e359229fb3 (127.2.81.67:42817)
I20260924 12:47:28.650055 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64baf1bc208d40349d75b234f3182f04" candidate_uuid: "3f832f99a8584630bc5110abbe4366dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28c72131a494f0b880d82f65f52cd70"
I20260924 12:47:28.650269 12870 raft_consensus.cc:3037] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:28.651361 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64baf1bc208d40349d75b234f3182f04" candidate_uuid: "3f832f99a8584630bc5110abbe4366dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f9e40dcb8a48e5942f78e359229fb3"
I20260924 12:47:28.651412 13138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64baf1bc208d40349d75b234f3182f04" candidate_uuid: "3f832f99a8584630bc5110abbe4366dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f9e40dcb8a48e5942f78e359229fb3" is_pre_election: true
I20260924 12:47:28.651888 13137 raft_consensus.cc:3037] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:47:28.653489 12870 raft_consensus.cc:2454] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f832f99a8584630bc5110abbe4366dc in term 1.
I20260924 12:47:28.654096 12939 leader_election.cc:302] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f832f99a8584630bc5110abbe4366dc, c28c72131a494f0b880d82f65f52cd70; no voters:
I20260924 12:47:28.654430 13208 raft_consensus.cc:2781] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:47:28.654923 13208 raft_consensus.cc:686] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 1 LEADER]: Becoming Leader. State: Replica: 3f832f99a8584630bc5110abbe4366dc, State: Running, Role: LEADER
I20260924 12:47:28.655215 13137 raft_consensus.cc:2454] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f832f99a8584630bc5110abbe4366dc in term 1.
I20260924 12:47:28.655476 13208 consensus_queue.cc:237] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } }
I20260924 12:47:28.660326 12725 catalog_manager.cc:5495] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f832f99a8584630bc5110abbe4366dc (127.2.81.66). New cstate: current_term: 1 leader_uuid: "3f832f99a8584630bc5110abbe4366dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f832f99a8584630bc5110abbe4366dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 } health_report { overall_health: UNKNOWN } } }
I20260924 12:47:28.682737 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.677322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.2.81.66:41011)
0924 12:47:28.677517 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.677521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.677540 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:47:28.677733 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.677737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.677814 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.677952 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:47:28.678315 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.679059 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.680490 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.680905 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.681049 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.681577 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:47:28.682132 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:47:28.682262 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.682394 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.682468 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 12:47:28.691354 12916 tablet.cc:2369] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:47:28.691627 13137 raft_consensus.cc:1261] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 1 FOLLOWER]: Refusing update from remote peer 3f832f99a8584630bc5110abbe4366dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:47:28.691744 12870 raft_consensus.cc:1261] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 1 FOLLOWER]: Refusing update from remote peer 3f832f99a8584630bc5110abbe4366dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:47:28.692737 13208 consensus_queue.cc:1035] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f9e40dcb8a48e5942f78e359229fb3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:47:28.693135 13213 consensus_queue.cc:1035] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c28c72131a494f0b880d82f65f52cd70" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:47:28.704141 13216 mvcc.cc:205] Tried to move back new op lower bound from 7332880583429640192 to 7332880583296991232. Current Snapshot: MvccSnapshot[applied={T|T < 7332880583429640192}]
I20260924 12:47:28.711714 13218 mvcc.cc:205] Tried to move back new op lower bound from 7332880583429640192 to 7332880583296991232. Current Snapshot: MvccSnapshot[applied={T|T < 7332880583429640192}]
I20260924 12:47:28.715094 13217 mvcc.cc:205] Tried to move back new op lower bound from 7332880583429640192 to 7332880583296991232. Current Snapshot: MvccSnapshot[applied={T|T < 7332880583429640192}]
I20260924 12:47:28.720460 12724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:47:28.720767 12724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:47:28.724606 12724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5af0308b644479890c19c8b0b513f5a: Sending DeleteTablet for 3 replicas of tablet 64baf1bc208d40349d75b234f3182f04
I20260924 12:47:28.726015 12850 tablet_service.cc:1514] Processing DeleteTablet for tablet 64baf1bc208d40349d75b234f3182f04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:47:28 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:48586
I20260924 12:47:28.726023 12984 tablet_service.cc:1514] Processing DeleteTablet for tablet 64baf1bc208d40349d75b234f3182f04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:47:28 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:41954
I20260924 12:47:28.726068 13118 tablet_service.cc:1514] Processing DeleteTablet for tablet 64baf1bc208d40349d75b234f3182f04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:47:28 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:38144
I20260924 12:47:28.729628 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12787
I20260924 12:47:28.739264 13228 tablet_replica.cc:330] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: stopping tablet replica
I20260924 12:47:28.739758 13228 raft_consensus.cc:2229] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:47:28.740327 13228 raft_consensus.cc:2258] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:47:28.746472 13228 ts_tablet_manager.cc:1905] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:47:28.749349 13226 tablet_replica.cc:330] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: stopping tablet replica
I20260924 12:47:28.752302 13226 raft_consensus.cc:2229] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:47:28.754809 13228 ts_tablet_manager.cc:1918] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:47:28.755013 13228 log.cc:1196] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/64baf1bc208d40349d75b234f3182f04
I20260924 12:47:28.755663 13228 ts_tablet_manager.cc:1939] T 64baf1bc208d40349d75b234f3182f04 P 84f9e40dcb8a48e5942f78e359229fb3: Deleting consensus metadata
I20260924 12:47:28.755832 13226 raft_consensus.cc:2258] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:47:28.756858 13227 tablet_replica.cc:330] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: stopping tablet replica
I20260924 12:47:28.757973 13227 raft_consensus.cc:2229] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 1 LEADER]: Raft consensus shutting down.
Sep 24 12:47:28 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:47:28.758862 13227 raft_consensus.cc:2258] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:47:28.759541 13226 ts_tablet_manager.cc:1905] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:47:28.760231 12711 catalog_manager.cc:4841] TS 84f9e40dcb8a48e5942f78e359229fb3 (127.2.81.67:42817): tablet 64baf1bc208d40349d75b234f3182f04 (table test-table [id=e427d2a61bac49379e5d890c095f5f4a]) successfully deleted
I20260924 12:47:28.762676 13227 ts_tablet_manager.cc:1905] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:47:28.767580 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.730863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.2.81.65:36745)
0924 12:47:28.731060 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.731067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.734972 (+ 3905us) server_negotiation.cc:407] Connection header received
0924 12:47:28.735964 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.735973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.736104 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.736208 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:28.740871 (+ 4663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.742252 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.749658 (+ 7406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.750444 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.750574 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.759588 (+ 9014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:28.759602 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:28.759619 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:28.759667 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:28.763577 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.764697 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.764708 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.764712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.764929 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.765475 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.765482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.765485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.765982 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:28.766304 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.766896 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.767164 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:47:28.767683 13226 ts_tablet_manager.cc:1918] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:47:28.768679 13226 log.cc:1196] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/64baf1bc208d40349d75b234f3182f04
I20260924 12:47:28.769352 12910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:28.769377 13226 ts_tablet_manager.cc:1939] T 64baf1bc208d40349d75b234f3182f04 P c28c72131a494f0b880d82f65f52cd70: Deleting consensus metadata
I20260924 12:47:28.770802 13227 ts_tablet_manager.cc:1918] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:47:28.771009 13227 log.cc:1196] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/64baf1bc208d40349d75b234f3182f04
I20260924 12:47:28.771683 13227 ts_tablet_manager.cc:1939] T 64baf1bc208d40349d75b234f3182f04 P 3f832f99a8584630bc5110abbe4366dc: Deleting consensus metadata
I20260924 12:47:28.774398 12710 catalog_manager.cc:4841] TS 3f832f99a8584630bc5110abbe4366dc (127.2.81.66:41011): tablet 64baf1bc208d40349d75b234f3182f04 (table test-table [id=e427d2a61bac49379e5d890c095f5f4a]) successfully deleted
I20260924 12:47:28.862203 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12787
I20260924 12:47:28.862267 12712 catalog_manager.cc:4841] TS c28c72131a494f0b880d82f65f52cd70 (127.2.81.65:36745): tablet 64baf1bc208d40349d75b234f3182f04 (table test-table [id=e427d2a61bac49379e5d890c095f5f4a]) successfully deleted
I20260924 12:47:28.876349 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12921
Sep 24 12:47:28 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:47:28.893632 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:28.877079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.2.81.66:41011)
0924 12:47:28.877246 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:28.877256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:28.877324 (+ 68us) server_negotiation.cc:407] Connection header received
0924 12:47:28.877507 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:28.877513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:28.877592 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:28.877678 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:28.878921 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.879834 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.881163 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:28.881597 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:28.881709 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:28.886112 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:28.886132 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:28.886151 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:28.886194 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:28.889355 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.890407 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.890420 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.890423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.890633 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:28.891136 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:28.891140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:28.891169 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:28.891556 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:28.891842 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:28.892773 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 12:47:28.893212 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:47:28.894642 13044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:29.009682 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12921
I20260924 12:47:29.024365 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 13055
Sep 24 12:47:29 dist-test-slave-9tm1 krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:47:29.039355 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:47:29.025095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.2.81.67:42817)
0924 12:47:29.025261 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 12:47:29.025266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:47:29.025285 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:47:29.025468 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:47:29.025473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:47:29.025550 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:47:29.025673 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:47:29.026850 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:29.027556 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:29.028803 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:47:29.029253 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:47:29.029364 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:47:29.033748 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:47:29.033763 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:47:29.033780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:47:29.033820 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:47:29.036446 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:29.037380 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:29.037388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:29.037390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:29.037533 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:47:29.037896 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:47:29.037900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:47:29.037902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:47:29.038260 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:47:29.038522 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 12:47:29.038838 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 12:47:29.039046 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 12:47:29.039960 13178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:29.129741 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 13055
I20260924 12:47:29.145589 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 12694
I20260924 12:47:29.147009 12756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:47:29.257129 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 12694
2026-09-24T12:47:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4152 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:47:29 dist-test-slave-9tm1 krb5kdc[13236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:47:29 dist-test-slave-9tm1 krb5kdc[13236](info): set up 2 sockets
Sep 24 12:47:29 dist-test-slave-9tm1 krb5kdc[13236](info): commencing operation
krb5kdc: starting...
W20260924 12:47:30.305900 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:47:30 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:47:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/rangeradmin_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/rangeradmin_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/rangerlookup_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/rangerlookup_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
I20260924 12:47:30.631523 2373 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYQebFB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/postgres -l logfile start
2026-09-24 12:47:33.600 UTC [13266] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 12:47:33.600 UTC [13266] LOG: listening on IPv4 address "127.2.81.84", port 39759
2026-09-24 12:47:33.605 UTC [13266] LOG: listening on Unix socket "/tmp/.s.PGSQL.39759"
2026-09-24 12:47:33.626 UTC [13269] LOG: database system was shut down at 2026-09-24 12:47:31 UTC
2026-09-24 12:47:33.630 UTC [13266] LOG: database system is ready to accept connections
I20260924 12:47:34.529665 2373 mini_postgres.cc:96] Postgres bound to 39759
2026-09-24 12:47:34.546 UTC [13277] FATAL: database "slave" does not exist
127.2.81.84:39759 - accepting connections
I20260924 12:47:34.547279 2373 mini_ranger.cc:161] Starting Ranger...
I20260924 12:47:34.572901 2373 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 12:47:35.183862 2373 mini_ranger.cc:87] Created ranger Postgres database
I20260924 12:47:35.183982 2373 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-admin
2026-09-24 12:47:35,476 [I] DB FLAVOR :POSTGRES
2026-09-24 12:47:35,476 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 12:47:35,476 [I] Checking connection..
2026-09-24 12:47:35,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 12:47:35,740 [I] Checking connection passed.
2026-09-24 12:47:35,740 [I] --------- Verifying version history table ---------
2026-09-24 12:47:35,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 12:47:36,011 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 12:47:36,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 12:47:36,257 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 12:47:36,258 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 12:47:36,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 12:47:36.502 UTC [13341] WARNING: there is no transaction in progress
2026-09-24 12:47:36,508 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 12:47:36,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 12:47:36,763 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 12:47:36,763 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 12:47:36,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 12:47:37,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 12:47:37,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tm1.c.gcp-upstream.internal','N') ;"
2026-09-24 12:47:37,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 12:47:37,801 [I] Table x_portal_user does not exist in database ranger
2026-09-24 12:47:37,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 12:47:38,097 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 12:47:38,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 12:47:38,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 12:47:38,609 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 12:47:38,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 12:47:39.807 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.826 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.843 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.861 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.882 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.902 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.973 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.981 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:39.997 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.013 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.025 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.041 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.055 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.064 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.073 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40.083 UTC [13449] WARNING: there is no transaction in progress
2026-09-24 12:47:40,885 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 12:47:40,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 12:47:41,128 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 12:47:41,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 12:47:41,401 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 12:47:41,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 12:47:41,683 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 12:47:41,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 12:47:41,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 12:47:42,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tm1.c.gcp-upstream.internal';"
2026-09-24 12:47:42,488 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 12:47:42,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 12:47:42,753 [I] DB_PATCHES have already been applied
I20260924 12:47:42.759325 2373 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYQebFB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 12:47:42.759421 2373 mini_ranger.cc:193] Using host: 127.2.81.84
I20260924 12:47:42.773977 2373 mini_ranger.cc:227] Ranger admin URL: http://127.2.81.84:57365
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:47:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 12:47:44.926113 2373 mini_ranger.cc:160] Time spent starting Ranger: real 10.379s user 0.009s sys 0.041s
Sep 24, 2026 12:48:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:48:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 12:48:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:48:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:48:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 12:48:07.274708 2373 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/rangerkms_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/rangerkms_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.81.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.81.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 12:48:07 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254087, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 12:48:07.462134 2373 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 12:48:07.486972 2373 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 12:48:08.614799 2373 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 12:48:08.614952 2373 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms
I20260924 12:48:08.615075 2373 mini_ranger_kms.cc:227] Using postgres at 127.2.81.84:39759
2026-09-24 12:48:09,004 [I] DB FLAVOR :POSTGRES
2026-09-24 12:48:09,004 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 12:48:09,004 [I] Checking connection
2026-09-24 12:48:09,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 12:48:09,267 [I] connection success
2026-09-24 12:48:09,267 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 12:48:09,268 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 12:48:09,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 12:48:09,562 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 12:48:09,562 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 12:48:09,563 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 12:48:09,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:39759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 12:48:09,850 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 12:48:10.415750 2373 mini_ranger_kms.cc:310] Created kms service
I20260924 12:48:10.605471 2373 mini_ranger_kms.cc:326] Created kudu user
I20260924 12:48:10.653079 2373 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 12:48:10 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254087, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:48:11.096233 2373 mini_ranger_kms.cc:384] Added ranger policy
I20260924 12:48:11.096535 2373 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 12:48:11.096603 2373 mini_ranger_kms.cc:240] Using host: 127.2.81.84
I20260924 12:48:11.117635 2373 mini_ranger_kms.cc:276] Ranger KMS PID: 13631
I20260924 12:48:11.117911 2373 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.81.84:39839
Sep 24, 2026 12:48:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:48:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:48:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:48:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:48:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:48:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 12:48:13.268944 2373 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.807s user 0.121s sys 0.291s
I20260924 12:48:13.269192 2373 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 12:48:13.269367 2373 mini_ranger_kms.cc:401] 127.2.81.84:39839/kms/v1/keys
Sep 24 12:48:13 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254093, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.81.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 12:48:13 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254093, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.81.84@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
Sep 24 12:48:14 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254093, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.81.84@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
Sep 24 12:48:16 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:48:17.013167 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39047
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39047
--ranger_config_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:39839/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:48:17.401571 13681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:17.401911 13681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:17.402045 13681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:17.411778 13681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:48:17.411924 13681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:48:17.411970 13681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:17.412019 13681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:48:17.412056 13681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:48:17.423520 13681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.81.84:39839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39047
--ranger_config_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:17.424911 13681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:17.426296 13681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:17.441323 13686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:17.441385 13689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:17.441355 13687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:17.441854 13681 server_base.cc:1034] running on GCE node
I20260924 12:48:17.442866 13681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:17.444398 13681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:17.445744 13681 hybrid_clock.cc:648] HybridClock initialized: now 1790254097445675 us; error 56 us; skew 500 ppm
I20260924 12:48:17.446280 13681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:48:17 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254097, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:48:17.454882 13681 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:48:17.457401 13681 webserver.cc:460] Webserver started at http://127.2.81.126:38591/ using document root <none> and password file <none>
I20260924 12:48:17.458441 13681 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:17.458757 13681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:17.459446 13681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:48:17 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254097, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:48:17.536139 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "bb0e301530cf48159ae1e8e8cad43278"
format_stamp: "Formatted at 2026-09-24 12:48:17 on dist-test-slave-9tm1"
server_key: "ba111f1f4d8465b7ccd6633976bcc92d"
server_key_iv: "51f7f44458781f95a6336b96b9dd22a4"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:17.537024 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "bb0e301530cf48159ae1e8e8cad43278"
format_stamp: "Formatted at 2026-09-24 12:48:17 on dist-test-slave-9tm1"
server_key: "ba111f1f4d8465b7ccd6633976bcc92d"
server_key_iv: "51f7f44458781f95a6336b96b9dd22a4"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:17.542972 13681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:48:17.565285 13696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:17.566923 13681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:48:17.567225 13681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "bb0e301530cf48159ae1e8e8cad43278"
format_stamp: "Formatted at 2026-09-24 12:48:17 on dist-test-slave-9tm1"
server_key: "ba111f1f4d8465b7ccd6633976bcc92d"
server_key_iv: "51f7f44458781f95a6336b96b9dd22a4"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:17.567456 13681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:17.592615 13681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:17.597257 13681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:17.597548 13681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:17.619781 13681 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:39047
I20260924 12:48:17.619789 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:39047 every 8 connection(s)
I20260924 12:48:17.622509 13681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:48:17.626940 13748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:17.627215 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 13681
I20260924 12:48:17.627408 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:48:17.628161 2373 external_mini_cluster.cc:1428] Setting key 903b353567ae4f9de6fc49135c96e307
I20260924 12:48:17.644259 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278: Bootstrap starting.
I20260924 12:48:17.648108 13748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:17.649797 13748 log.cc:824] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:17.654078 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278: No bootstrap required, opened a new log
Sep 24 12:48:17 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:48:17.662997 13748 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0e301530cf48159ae1e8e8cad43278" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39047 } }
I20260924 12:48:17.663434 13748 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:17.663545 13748 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0e301530cf48159ae1e8e8cad43278, State: Initialized, Role: FOLLOWER
I20260924 12:48:17.664338 13748 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0e301530cf48159ae1e8e8cad43278" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39047 } }
I20260924 12:48:17.664601 13748 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:48:17.664757 13748 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:48:17.664960 13748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:17.668473 13748 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0e301530cf48159ae1e8e8cad43278" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39047 } }
I20260924 12:48:17.669123 13748 leader_election.cc:302] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0e301530cf48159ae1e8e8cad43278; no voters:
I20260924 12:48:17.672235 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:17.630634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.2.81.126:39047)
0924 12:48:17.645163 (+ 14529us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:17.645182 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:17.645719 (+ 537us) server_negotiation.cc:407] Connection header received
0924 12:48:17.648376 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:17.648410 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:17.649073 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:17.649563 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:17.651134 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:17.653584 (+ 2450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:17.655742 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:17.657089 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:17.657331 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:17.662346 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:17.662359 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:17.662384 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:17.662444 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:17.666518 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:17.667523 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:17.667533 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:17.667543 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:17.667753 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:17.668209 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:17.668214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:17.668217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:17.669011 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:17.669307 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:17.669984 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 12:48:17.670540 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14181,"thread_start_us":14448,"threads_started":1}
I20260924 12:48:17.673286 13748 leader_election.cc:288] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:48:17.673398 13753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:17.677664 13753 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [term 1 LEADER]: Becoming Leader. State: Replica: bb0e301530cf48159ae1e8e8cad43278, State: Running, Role: LEADER
I20260924 12:48:17.678490 13753 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0e301530cf48159ae1e8e8cad43278" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39047 } }
I20260924 12:48:17.680842 13753 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0e301530cf48159ae1e8e8cad43278. Latest consensus state: current_term: 1 leader_uuid: "bb0e301530cf48159ae1e8e8cad43278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0e301530cf48159ae1e8e8cad43278" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39047 } } }
I20260924 12:48:17.681300 13753 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:17.681571 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0e301530cf48159ae1e8e8cad43278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0e301530cf48159ae1e8e8cad43278" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39047 } } }
I20260924 12:48:17.682063 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [sys.catalog]: This master's current role is: LEADER
I20260924 12:48:17.684034 13748 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:48:17.688212 13757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:48:17.694725 13757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:48:17.717280 13757 catalog_manager.cc:1347] Generated new cluster ID: c3941c71f3bf445ea16a9a9215578388
I20260924 12:48:17.717407 13757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:48:17.728616 13757 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:48:17.730572 13757 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:48:17.741623 13757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278: Generated new TSK 0
I20260924 12:48:17.742836 13757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:48:17.755023 13748 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:48:18.674897 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:39839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39047
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:48:18 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254098, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 12:48:19.135751 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:19.136184 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:19.136404 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:19.148888 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:48:19.149087 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:19.149353 13793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:48:19.167675 13793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.81.84:39839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:19.169646 13793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:19.172104 13793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:19.195741 13803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:19.202387 13804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:19.203143 13806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:19.204500 13793 server_base.cc:1034] running on GCE node
I20260924 12:48:19.205500 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:19.207664 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:19.209203 13793 hybrid_clock.cc:648] HybridClock initialized: now 1790254099209073 us; error 69 us; skew 500 ppm
I20260924 12:48:19.209685 13793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:48:19 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254099, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:48:19.215767 13793 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:48:19.218616 13793 webserver.cc:460] Webserver started at http://127.2.81.65:40447/ using document root <none> and password file <none>
I20260924 12:48:19.219736 13793 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:19.219910 13793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:19.220366 13793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:48:19 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254099, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:48:19.244038 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "c4df707611444e99b09691717432000e"
format_stamp: "Formatted at 2026-09-24 12:48:19 on dist-test-slave-9tm1"
server_key: "675a1bc90c8584cb3c4acab8df6fabbd"
server_key_iv: "6fb74b464c310c4d91839975661c1062"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:19.246291 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "c4df707611444e99b09691717432000e"
format_stamp: "Formatted at 2026-09-24 12:48:19 on dist-test-slave-9tm1"
server_key: "675a1bc90c8584cb3c4acab8df6fabbd"
server_key_iv: "6fb74b464c310c4d91839975661c1062"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:19.256287 13793 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260924 12:48:19.277027 13813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:19.278666 13793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 12:48:19.278867 13793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "c4df707611444e99b09691717432000e"
format_stamp: "Formatted at 2026-09-24 12:48:19 on dist-test-slave-9tm1"
server_key: "675a1bc90c8584cb3c4acab8df6fabbd"
server_key_iv: "6fb74b464c310c4d91839975661c1062"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:19.279099 13793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:19.301247 13793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:19.309147 13793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:19.309628 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:19.311641 13793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:19.313977 13793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:19.314097 13793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:19.314311 13793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:19.314386 13793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:19.403105 13793 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:35541
I20260924 12:48:19.403110 13925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:35541 every 8 connection(s)
I20260924 12:48:19.405062 13793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:48:19.406031 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 13793
I20260924 12:48:19.406363 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:48:19.407891 2373 external_mini_cluster.cc:1428] Setting key 4d7031e326afaee11660e092f5458197
Sep 24 12:48:19 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254099, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:48:19.457538 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:19.408957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:36727 (local address 127.2.81.126:39047)
0924 12:48:19.430415 (+ 21458us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:19.430422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:19.430452 (+ 30us) server_negotiation.cc:407] Connection header received
0924 12:48:19.432067 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:19.432074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:19.432205 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:19.432355 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:19.434843 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:19.436136 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:19.438818 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:19.439527 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:19.439701 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:19.445273 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:19.445281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:19.445287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:19.445339 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:19.451247 (+ 5908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:19.453466 (+ 2219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:19.453475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:19.453480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:19.453795 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:19.454872 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:19.454882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:19.454886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:19.455365 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:19.455539 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:19.456814 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0924 12:48:19.457130 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21268,"thread_start_us":21468,"threads_started":1}
I20260924 12:48:19.460268 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:19.412469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39047 (local address 127.2.81.65:36727)
0924 12:48:19.426560 (+ 14091us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:19.426654 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:19.430887 (+ 4233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:19.432720 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:19.432735 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:19.433467 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:48:19.434559 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:19.434593 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:19.436392 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:19.436399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:19.438583 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:19.438618 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:19.439838 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:19.439846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:19.440690 (+ 844us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:19.441606 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:48:19.441643 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:48:19.444816 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:48:19.451839 (+ 7023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:48:19.452007 (+ 168us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:48:19.452030 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:48:19.453072 (+ 1042us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:48:19.454189 (+ 1117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:48:19.454195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:48:19.454199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:48:19.454556 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:48:19.455733 (+ 1177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:48:19.455745 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:48:19.456267 (+ 522us) client_negotiation.cc:769] Sending connection context
0924 12:48:19.458079 (+ 1812us) client_negotiation.cc:240] Negotiation successful
0924 12:48:19.458760 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13573,"thread_start_us":13688,"threads_started":1}
I20260924 12:48:19.462648 13926 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39047
I20260924 12:48:19.463002 13926 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:19.463802 13926 heartbeater.cc:511] Master 127.2.81.126:39047 requested a full tablet report, sending...
I20260924 12:48:19.466863 13713 ts_manager.cc:194] Registered new tserver with Master: c4df707611444e99b09691717432000e (127.2.81.65:35541)
I20260924 12:48:19.469241 13713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:36727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:48:19.505475 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:39839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39047
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:48:19 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254098, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
Sep 24 12:48:19 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254098, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
W20260924 12:48:20.068019 13932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:20.068547 13932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:20.068756 13932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:20.083187 13932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:48:20.084411 13932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:20.084717 13932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:48:20.102180 13932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.81.84:39839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:20.104219 13932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:20.106276 13932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:20.126602 13940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:20.127060 13942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:20.127113 13939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:20.127597 13932 server_base.cc:1034] running on GCE node
I20260924 12:48:20.128469 13932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:20.131274 13932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:20.132601 13932 hybrid_clock.cc:648] HybridClock initialized: now 1790254100132530 us; error 76 us; skew 500 ppm
I20260924 12:48:20.133116 13932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:48:20 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:48:20.138370 13932 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:48:20.142267 13932 webserver.cc:460] Webserver started at http://127.2.81.66:45047/ using document root <none> and password file <none>
I20260924 12:48:20.143386 13932 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:20.143597 13932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:20.144027 13932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:48:20 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:48:20.164397 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "731c04e236e44501b70e1b8eba9a59dc"
format_stamp: "Formatted at 2026-09-24 12:48:20 on dist-test-slave-9tm1"
server_key: "5041ad92faf6397400b25dcadc816142"
server_key_iv: "03471a4ae0580da11385f2c37818d956"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:20.165494 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "731c04e236e44501b70e1b8eba9a59dc"
format_stamp: "Formatted at 2026-09-24 12:48:20 on dist-test-slave-9tm1"
server_key: "5041ad92faf6397400b25dcadc816142"
server_key_iv: "03471a4ae0580da11385f2c37818d956"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:20.182379 13932 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.009s sys 0.004s
I20260924 12:48:20.210911 13952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:20.214706 13932 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.006s
I20260924 12:48:20.214960 13932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "731c04e236e44501b70e1b8eba9a59dc"
format_stamp: "Formatted at 2026-09-24 12:48:20 on dist-test-slave-9tm1"
server_key: "5041ad92faf6397400b25dcadc816142"
server_key_iv: "03471a4ae0580da11385f2c37818d956"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:20.215257 13932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:20.268927 13932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:20.277629 13932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:20.278048 13932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:20.279461 13932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:20.281595 13932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:20.281711 13932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:20.281896 13932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:20.281987 13932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:20.342339 13932 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:33245
I20260924 12:48:20.342446 14068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:33245 every 8 connection(s)
I20260924 12:48:20.344115 13932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:48:20.345852 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 13932
I20260924 12:48:20.346021 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:48:20.346577 2373 external_mini_cluster.cc:1428] Setting key 7a6b87b8d0dc135e2a9877e0f6ab4b68
Sep 24 12:48:20 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:48:20.387501 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:20.347951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:50407 (local address 127.2.81.126:39047)
0924 12:48:20.361509 (+ 13558us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:20.361516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:20.361539 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:48:20.361604 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:20.361609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:20.361716 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:20.361839 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:20.364146 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:20.365286 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:20.368329 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:20.369007 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:20.369161 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:20.376450 (+ 7289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:20.376459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:20.376464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:20.376518 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:20.380648 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:20.382002 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:20.382008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:20.382012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:20.382182 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:20.382829 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:20.382836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:20.382840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:20.383259 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:20.383390 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:20.386746 (+ 3356us) server_negotiation.cc:299] Negotiation successful
0924 12:48:20.387101 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13408,"thread_start_us":15964,"threads_started":1}
I20260924 12:48:20.388469 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:20.348136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39047 (local address 127.2.81.66:50407)
0924 12:48:20.357327 (+ 9191us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:20.357401 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:20.359765 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:20.362323 (+ 2558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:20.362339 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:20.363033 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:48:20.363897 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:20.363924 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:20.366083 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:20.366088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:20.368116 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:20.368136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:20.371122 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:20.371134 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:20.371488 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:20.372426 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:48:20.372460 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:48:20.376078 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:48:20.380925 (+ 4847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:48:20.380938 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:48:20.380960 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:48:20.381812 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:48:20.382324 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:48:20.382329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:48:20.382333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:48:20.382613 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:48:20.383477 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:48:20.383486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:48:20.384059 (+ 573us) client_negotiation.cc:769] Sending connection context
0924 12:48:20.384495 (+ 436us) client_negotiation.cc:240] Negotiation successful
0924 12:48:20.385135 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8777,"thread_start_us":11997,"threads_started":1}
I20260924 12:48:20.390551 14069 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39047
I20260924 12:48:20.390919 14069 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
I20260924 12:48:20.391808 14069 heartbeater.cc:511] Master 127.2.81.126:39047 requested a full tablet report, sending...
I20260924 12:48:20.393725 13713 ts_manager.cc:194] Registered new tserver with Master: 731c04e236e44501b70e1b8eba9a59dc (127.2.81.66:33245)
I20260924 12:48:20.394809 13713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:50407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:48:20.428884 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:39839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39047
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:48:20.473399 13926 heartbeater.cc:503] Master 127.2.81.126:39047 was elected leader, sending a full tablet report...
W20260924 12:48:20.863359 14075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:48:20.863660 14075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:48:20.863807 14075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:48:20.873507 14075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:48:20.873697 14075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:48:20.873817 14075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:48:20.887111 14075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.81.84:39839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:48:20.888502 14075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:48:20.889998 14075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:48:20.903807 14081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:20.903867 14083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:48:20.903892 14080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:48:20.905187 14075 server_base.cc:1034] running on GCE node
I20260924 12:48:20.905809 14075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:48:20.906836 14075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:48:20.908092 14075 hybrid_clock.cc:648] HybridClock initialized: now 1790254100907993 us; error 62 us; skew 500 ppm
I20260924 12:48:20.908450 14075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:48:20 dist-test-slave-9tm1 krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:48:20.911744 14075 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:48:20.913479 14075 webserver.cc:460] Webserver started at http://127.2.81.67:33009/ using document root <none> and password file <none>
I20260924 12:48:20.914171 14075 fs_manager.cc:359] Metadata directory not provided
I20260924 12:48:20.914283 14075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:48:20.914561 14075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:48:20 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:48:20.931324 14075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "d87bf98b6c024e569deec54e73a02265"
format_stamp: "Formatted at 2026-09-24 12:48:20 on dist-test-slave-9tm1"
server_key: "0d700bd4369174b4921a7c2eb35adaf3"
server_key_iv: "de79d88ce6f0025a1c74799fa5217cef"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:20.932212 14075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "d87bf98b6c024e569deec54e73a02265"
format_stamp: "Formatted at 2026-09-24 12:48:20 on dist-test-slave-9tm1"
server_key: "0d700bd4369174b4921a7c2eb35adaf3"
server_key_iv: "de79d88ce6f0025a1c74799fa5217cef"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:20.937943 14075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 12:48:20.958488 14090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:20.960311 14075 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260924 12:48:20.960518 14075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "d87bf98b6c024e569deec54e73a02265"
format_stamp: "Formatted at 2026-09-24 12:48:20 on dist-test-slave-9tm1"
server_key: "0d700bd4369174b4921a7c2eb35adaf3"
server_key_iv: "de79d88ce6f0025a1c74799fa5217cef"
server_key_version: "kuduclusterkey@0"
I20260924 12:48:20.960865 14075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:48:20.982604 14075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:48:20.987188 14075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:48:20.987450 14075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:48:20.988523 14075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:48:20.989926 14075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:48:20.990008 14075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:20.990094 14075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:48:20.990136 14075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:48:21.027555 14075 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:43405
I20260924 12:48:21.027588 14202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:43405 every 8 connection(s)
I20260924 12:48:21.028843 14075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:48:21.037601 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 14075
I20260924 12:48:21.037794 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:48:21.038254 2373 external_mini_cluster.cc:1428] Setting key 275a21fe1cbb5e9eb83056049970f0d9
Sep 24 12:48:21 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:48:21.052374 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.032357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:43271 (local address 127.2.81.126:39047)
0924 12:48:21.032862 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.032866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.033995 (+ 1129us) server_negotiation.cc:407] Connection header received
0924 12:48:21.035968 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.035975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.036082 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.036185 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:21.038033 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.038850 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.041197 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.041651 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.041761 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.045609 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:21.045616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:21.045621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:21.045660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:21.048347 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.049405 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.049410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.049413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.049565 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.050533 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.050539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.050541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.050837 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:21.050975 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.051802 (+ 827us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.052032 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260924 12:48:21.053746 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.032931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39047 (local address 127.2.81.67:43271)
0924 12:48:21.033805 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:21.033850 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:21.035635 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:21.036469 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:21.036479 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:21.036929 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:48:21.037805 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.037836 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.039263 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.039267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.041000 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.041019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.041865 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.041871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.042135 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.042815 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:48:21.042841 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:48:21.045297 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:48:21.048555 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:48:21.048565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:48:21.048581 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:48:21.049221 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:48:21.049704 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:48:21.049708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:48:21.049710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:48:21.050362 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:48:21.050975 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:48:21.050984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:48:21.051338 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 12:48:21.051640 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 12:48:21.052131 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260924 12:48:21.055452 14203 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39047
I20260924 12:48:21.055900 14203 heartbeater.cc:464] Registering TS with master...
I20260924 12:48:21.056682 14203 heartbeater.cc:511] Master 127.2.81.126:39047 requested a full tablet report, sending...
I20260924 12:48:21.058373 13713 ts_manager.cc:194] Registered new tserver with Master: d87bf98b6c024e569deec54e73a02265 (127.2.81.67:43405)
I20260924 12:48:21.059310 13713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:43271
I20260924 12:48:21.067315 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:48:21.087280 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.072123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.2.81.126:39047)
0924 12:48:21.072301 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.072306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.072767 (+ 461us) server_negotiation.cc:407] Connection header received
0924 12:48:21.072929 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.072936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.073062 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.073157 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:21.074464 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.075196 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.076641 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.077209 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.077316 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.079853 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:21.079946 (+ 93us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:21.079959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:21.080010 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:21.082865 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.083792 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.083796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.083799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.083949 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.084604 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.084610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.084613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.085391 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:21.085638 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.086569 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.086871 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 12:48:21.093111 13713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:48:21.100805 13713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:48:21.131891 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.118308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.2.81.66:33245)
0924 12:48:21.122009 (+ 3701us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.122019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.124411 (+ 2392us) server_negotiation.cc:407] Connection header received
0924 12:48:21.124507 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.124512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.124673 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.124830 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:48:21.125399 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.127314 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.129276 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.130670 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.130798 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.131016 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.131399 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.131571 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3521,"thread_start_us":3413,"threads_started":1}
I20260924 12:48:21.131943 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.116690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:33245 (local address 127.0.0.1:54334)
0924 12:48:21.124207 (+ 7517us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:21.124224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:21.124325 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:21.124832 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:21.124836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:21.124857 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:48:21.125224 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.125235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.127450 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.127455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.129052 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.129064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.130911 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.130918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.131150 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.131193 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 12:48:21.131311 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 12:48:21.131533 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7380,"mutex_wait_us":49,"thread_start_us":5551,"threads_started":1}
I20260924 12:48:21.132478 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.116662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.2.81.65:35541)
0924 12:48:21.119499 (+ 2837us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.119510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.123059 (+ 3549us) server_negotiation.cc:407] Connection header received
0924 12:48:21.123176 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.123181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.123380 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.123545 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:48:21.124291 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.125956 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.129362 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.130867 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.130980 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.131326 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.131934 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.132127 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2631,"thread_start_us":2458,"threads_started":1}
I20260924 12:48:21.133244 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.116361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:35541 (local address 127.0.0.1:40346)
0924 12:48:21.122685 (+ 6324us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:21.122730 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:21.122910 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:21.123571 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:21.123577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:21.123623 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:48:21.124092 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.124110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.126584 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.126589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.129150 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.129165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.131550 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.131556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.131717 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.131749 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 12:48:21.132712 (+ 963us) client_negotiation.cc:240] Negotiation successful
0924 12:48:21.132856 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6075,"mutex_wait_us":66,"thread_start_us":7545,"threads_started":1}
I20260924 12:48:21.135586 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.123212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.2.81.67:43405)
0924 12:48:21.125492 (+ 2280us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.125501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.126430 (+ 929us) server_negotiation.cc:407] Connection header received
0924 12:48:21.126648 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.126655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.126831 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.126992 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:48:21.127652 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.129178 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.131659 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.133304 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.134275 (+ 971us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.134521 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.135052 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.135226 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2108,"thread_start_us":1953,"threads_started":1}
I20260924 12:48:21.135982 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.121992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43405 (local address 127.0.0.1:37490)
0924 12:48:21.126318 (+ 4326us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:21.126336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:21.126451 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:21.127005 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:21.127010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:21.127032 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:48:21.127423 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.127436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.129324 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.129329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.131426 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.131440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.134695 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.134701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.134871 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.134902 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:48:21.134994 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 12:48:21.135130 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4201,"thread_start_us":3639,"threads_started":1}
I20260924 12:48:21.136674 14004 tablet_service.cc:1467] Processing CreateTablet for tablet ff4536bd13f14ac29639a222685b564d (DEFAULT_TABLE table=test-table [id=8552cfe7a9364afd8215d49190884426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:21.136677 13861 tablet_service.cc:1467] Processing CreateTablet for tablet ff4536bd13f14ac29639a222685b564d (DEFAULT_TABLE table=test-table [id=8552cfe7a9364afd8215d49190884426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:21.138228 14004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4536bd13f14ac29639a222685b564d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:21.138221 13861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4536bd13f14ac29639a222685b564d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:21.144311 14138 tablet_service.cc:1467] Processing CreateTablet for tablet ff4536bd13f14ac29639a222685b564d (DEFAULT_TABLE table=test-table [id=8552cfe7a9364afd8215d49190884426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:48:21.145888 14138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4536bd13f14ac29639a222685b564d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:48:21.154871 14222 tablet_bootstrap.cc:492] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Bootstrap starting.
I20260924 12:48:21.155671 14223 tablet_bootstrap.cc:492] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Bootstrap starting.
I20260924 12:48:21.159466 14224 tablet_bootstrap.cc:492] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Bootstrap starting.
I20260924 12:48:21.159945 14223 tablet_bootstrap.cc:654] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:21.160640 14222 tablet_bootstrap.cc:654] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:21.161118 14223 log.cc:824] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:21.161885 14222 log.cc:824] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:21.162999 14224 tablet_bootstrap.cc:654] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Neither blocks nor log segments found. Creating new log.
I20260924 12:48:21.164417 14223 tablet_bootstrap.cc:492] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: No bootstrap required, opened a new log
I20260924 12:48:21.164683 14223 ts_tablet_manager.cc:1397] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 12:48:21.164883 14224 log.cc:824] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Log is configured to *not* fsync() on all Append() calls
I20260924 12:48:21.167680 14222 tablet_bootstrap.cc:492] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: No bootstrap required, opened a new log
I20260924 12:48:21.167979 14222 ts_tablet_manager.cc:1397] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260924 12:48:21.168066 14224 tablet_bootstrap.cc:492] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: No bootstrap required, opened a new log
I20260924 12:48:21.168355 14224 ts_tablet_manager.cc:1397] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 12:48:21.176133 14223 raft_consensus.cc:348] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.176694 14223 raft_consensus.cc:374] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:21.176838 14223 raft_consensus.cc:729] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731c04e236e44501b70e1b8eba9a59dc, State: Initialized, Role: FOLLOWER
I20260924 12:48:21.177167 14222 raft_consensus.cc:348] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.177577 14222 raft_consensus.cc:374] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:21.177687 14222 raft_consensus.cc:729] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4df707611444e99b09691717432000e, State: Initialized, Role: FOLLOWER
I20260924 12:48:21.177584 14223 consensus_queue.cc:260] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.178282 14222 consensus_queue.cc:260] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.178805 14069 heartbeater.cc:503] Master 127.2.81.126:39047 was elected leader, sending a full tablet report...
I20260924 12:48:21.178952 14224 raft_consensus.cc:348] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.179378 14224 raft_consensus.cc:374] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:48:21.179464 14224 raft_consensus.cc:729] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87bf98b6c024e569deec54e73a02265, State: Initialized, Role: FOLLOWER
I20260924 12:48:21.179987 14223 ts_tablet_manager.cc:1428] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 12:48:21.180092 14224 consensus_queue.cc:260] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.182806 14222 ts_tablet_manager.cc:1428] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260924 12:48:21.184765 14224 ts_tablet_manager.cc:1428] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
W20260924 12:48:21.185415 13927 tablet.cc:2369] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:21.186336 14203 heartbeater.cc:503] Master 127.2.81.126:39047 was elected leader, sending a full tablet report...
I20260924 12:48:21.243320 14229 raft_consensus.cc:482] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:48:21.243620 14229 raft_consensus.cc:504] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.245322 14229 leader_election.cc:288] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731c04e236e44501b70e1b8eba9a59dc (127.2.81.66:33245), d87bf98b6c024e569deec54e73a02265 (127.2.81.67:43405)
I20260924 12:48:21.253876 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.245762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:39569 (local address 127.2.81.66:33245)
0924 12:48:21.245970 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.245976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.246956 (+ 980us) server_negotiation.cc:407] Connection header received
0924 12:48:21.247158 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.247163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.247251 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.247335 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:48:21.247952 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.248754 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.251593 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.252820 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.253028 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.253137 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.253544 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.253681 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:48:21.254170 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.245660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:33245 (local address 127.2.81.65:39569)
0924 12:48:21.246840 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:21.246858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:21.246992 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:21.247373 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:21.247377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:21.247403 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:48:21.247750 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.247760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.248881 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.248886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.251361 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.251381 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.253200 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.253207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.253369 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.253398 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 12:48:21.253495 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 12:48:21.253657 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":71,"thread_start_us":890,"threads_started":1}
I20260924 12:48:21.254429 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.245762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:50103 (local address 127.2.81.67:43405)
0924 12:48:21.245969 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.245976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.246571 (+ 595us) server_negotiation.cc:407] Connection header received
0924 12:48:21.246903 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.246950 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.247029 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.247120 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:48:21.248174 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.248967 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.252086 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.253372 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.253495 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.253589 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.254076 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.254220 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:48:21.255201 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.245598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43405 (local address 127.2.81.65:50103)
0924 12:48:21.246468 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 12:48:21.246489 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:48:21.246726 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:48:21.247483 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:48:21.247487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:48:21.247528 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:48:21.248016 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.248030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.249133 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.249138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.251361 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:48:21.251381 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.253594 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.253600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:48:21.253780 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.253818 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:48:21.254597 (+ 779us) client_negotiation.cc:240] Negotiation successful
0924 12:48:21.254745 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":95,"thread_start_us":548,"threads_started":1}
I20260924 12:48:21.255403 14024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4536bd13f14ac29639a222685b564d" candidate_uuid: "c4df707611444e99b09691717432000e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731c04e236e44501b70e1b8eba9a59dc" is_pre_election: true
I20260924 12:48:21.255939 14024 raft_consensus.cc:2454] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4df707611444e99b09691717432000e in term 0.
I20260924 12:48:21.256201 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4536bd13f14ac29639a222685b564d" candidate_uuid: "c4df707611444e99b09691717432000e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87bf98b6c024e569deec54e73a02265" is_pre_election: true
I20260924 12:48:21.256633 13817 leader_election.cc:302] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731c04e236e44501b70e1b8eba9a59dc, c4df707611444e99b09691717432000e; no voters:
I20260924 12:48:21.256680 14158 raft_consensus.cc:2454] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4df707611444e99b09691717432000e in term 0.
I20260924 12:48:21.257062 14229 raft_consensus.cc:2781] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:48:21.257251 14229 raft_consensus.cc:482] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:48:21.257368 14229 raft_consensus.cc:3037] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:21.259953 14229 raft_consensus.cc:504] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.260780 14229 leader_election.cc:288] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [CANDIDATE]: Term 1 election: Requested vote from peers 731c04e236e44501b70e1b8eba9a59dc (127.2.81.66:33245), d87bf98b6c024e569deec54e73a02265 (127.2.81.67:43405)
I20260924 12:48:21.261428 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4536bd13f14ac29639a222685b564d" candidate_uuid: "c4df707611444e99b09691717432000e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87bf98b6c024e569deec54e73a02265"
I20260924 12:48:21.261471 14024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4536bd13f14ac29639a222685b564d" candidate_uuid: "c4df707611444e99b09691717432000e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731c04e236e44501b70e1b8eba9a59dc"
I20260924 12:48:21.261647 14158 raft_consensus.cc:3037] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:21.261729 14024 raft_consensus.cc:3037] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:48:21.264014 14158 raft_consensus.cc:2454] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4df707611444e99b09691717432000e in term 1.
I20260924 12:48:21.264223 14024 raft_consensus.cc:2454] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4df707611444e99b09691717432000e in term 1.
I20260924 12:48:21.264557 13815 leader_election.cc:302] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4df707611444e99b09691717432000e, d87bf98b6c024e569deec54e73a02265; no voters:
I20260924 12:48:21.264906 14229 raft_consensus.cc:2781] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:48:21.265400 14229 raft_consensus.cc:686] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 1 LEADER]: Becoming Leader. State: Replica: c4df707611444e99b09691717432000e, State: Running, Role: LEADER
I20260924 12:48:21.265884 14229 consensus_queue.cc:237] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } }
I20260924 12:48:21.271355 13712 catalog_manager.cc:5495] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e reported cstate change: term changed from 0 to 1, leader changed from <none> to c4df707611444e99b09691717432000e (127.2.81.65). New cstate: current_term: 1 leader_uuid: "c4df707611444e99b09691717432000e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4df707611444e99b09691717432000e" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 35541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 } health_report { overall_health: UNKNOWN } } }
W20260924 12:48:21.280896 14204 tablet.cc:2369] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:48:21.291294 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.285364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.2.81.65:35541)
0924 12:48:21.285588 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.285594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.285615 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:48:21.285938 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.285944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.286041 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.286210 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:48:21.286629 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.287407 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.288833 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.289299 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.289418 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.289953 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:48:21.290500 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:48:21.290788 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.290923 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.291016 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:48:21.301723 14024 raft_consensus.cc:1261] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 1 FOLLOWER]: Refusing update from remote peer c4df707611444e99b09691717432000e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:21.301760 14158 raft_consensus.cc:1261] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 1 FOLLOWER]: Refusing update from remote peer c4df707611444e99b09691717432000e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:48:21.302889 14233 consensus_queue.cc:1035] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [LEADER]: Connected to new peer: Peer: permanent_uuid: "731c04e236e44501b70e1b8eba9a59dc" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:21.303278 14229 consensus_queue.cc:1035] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87bf98b6c024e569deec54e73a02265" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:48:21.318233 14236 mvcc.cc:205] Tried to move back new op lower bound from 7332880798918586368 to 7332880798789017600. Current Snapshot: MvccSnapshot[applied={T|T < 7332880798918586368}]
I20260924 12:48:21.340080 13713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:48:21.340371 13713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:48:21.344504 13713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0e301530cf48159ae1e8e8cad43278: Sending DeleteTablet for 3 replicas of tablet ff4536bd13f14ac29639a222685b564d
I20260924 12:48:21.346091 14004 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4536bd13f14ac29639a222685b564d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:48:21 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:54334
I20260924 12:48:21.346206 13861 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4536bd13f14ac29639a222685b564d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:48:21 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:40346
I20260924 12:48:21.347129 14138 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4536bd13f14ac29639a222685b564d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:48:21 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:37490
I20260924 12:48:21.357203 14246 tablet_replica.cc:330] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: stopping tablet replica
I20260924 12:48:21.357897 14247 tablet_replica.cc:330] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: stopping tablet replica
I20260924 12:48:21.358577 14247 raft_consensus.cc:2229] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:48:21.359179 14246 raft_consensus.cc:2229] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:21.359431 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 13793
I20260924 12:48:21.360024 14246 raft_consensus.cc:2258] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:21.360630 14247 raft_consensus.cc:2258] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:21.363641 14247 ts_tablet_manager.cc:1905] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:48:21.363689 14248 tablet_replica.cc:330] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: stopping tablet replica
I20260924 12:48:21.364101 14248 raft_consensus.cc:2229] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:48:21.364600 14248 raft_consensus.cc:2258] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:48:21.365510 14246 ts_tablet_manager.cc:1905] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:48:21.372613 14248 ts_tablet_manager.cc:1905] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:48:21.372922 14247 ts_tablet_manager.cc:1918] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:48:21.373234 14247 log.cc:1196] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/ff4536bd13f14ac29639a222685b564d
I20260924 12:48:21.373466 14246 ts_tablet_manager.cc:1918] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:48:21.373752 14247 ts_tablet_manager.cc:1939] T ff4536bd13f14ac29639a222685b564d P c4df707611444e99b09691717432000e: Deleting consensus metadata
I20260924 12:48:21.373787 14246 log.cc:1196] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/ff4536bd13f14ac29639a222685b564d
I20260924 12:48:21.374382 14246 ts_tablet_manager.cc:1939] T ff4536bd13f14ac29639a222685b564d P 731c04e236e44501b70e1b8eba9a59dc: Deleting consensus metadata
I20260924 12:48:21.375696 13699 catalog_manager.cc:4841] TS c4df707611444e99b09691717432000e (127.2.81.65:35541): tablet ff4536bd13f14ac29639a222685b564d (table test-table [id=8552cfe7a9364afd8215d49190884426]) successfully deleted
I20260924 12:48:21.376166 13700 catalog_manager.cc:4841] TS 731c04e236e44501b70e1b8eba9a59dc (127.2.81.66:33245): tablet ff4536bd13f14ac29639a222685b564d (table test-table [id=8552cfe7a9364afd8215d49190884426]) successfully deleted
I20260924 12:48:21.379621 14248 ts_tablet_manager.cc:1918] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:48:21.379809 14248 log.cc:1196] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/ff4536bd13f14ac29639a222685b564d
I20260924 12:48:21.380306 14248 ts_tablet_manager.cc:1939] T ff4536bd13f14ac29639a222685b564d P d87bf98b6c024e569deec54e73a02265: Deleting consensus metadata
I20260924 12:48:21.381929 13698 catalog_manager.cc:4841] TS d87bf98b6c024e569deec54e73a02265 (127.2.81.67:43405): tablet ff4536bd13f14ac29639a222685b564d (table test-table [id=8552cfe7a9364afd8215d49190884426]) successfully deleted
Sep 24 12:48:21 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:48:21.397761 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.362266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.2.81.65:35541)
0924 12:48:21.362714 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.362720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.378147 (+ 15427us) server_negotiation.cc:407] Connection header received
0924 12:48:21.378354 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.378360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.378458 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.378574 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:21.380254 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.382059 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.383861 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.384619 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.384781 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.391121 (+ 6340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:21.391140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:21.391157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:21.391204 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:21.394366 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.395342 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.395350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.395353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.395519 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.395977 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.395981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.395983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.396426 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:21.396668 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.397200 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.397422 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260924 12:48:21.398458 13921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:21.501479 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 13793
I20260924 12:48:21.517567 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 13932
Sep 24 12:48:21 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:48:21.536191 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.518378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.2.81.66:33245)
0924 12:48:21.518550 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.518554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.518573 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:48:21.518779 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.518783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.518862 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.518953 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:21.520343 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.521409 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.523308 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.523899 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.524052 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.529023 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:21.529046 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:21.529063 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:21.529112 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:21.532410 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.533876 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.533885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.533887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.534074 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.534513 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.534517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.534520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.534929 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:21.535228 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.535560 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.535796 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:48:21.536883 14064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:21.636500 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 13932
I20260924 12:48:21.651484 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 14075
Sep 24 12:48:21 dist-test-slave-9tm1 krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:48:21.671727 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:48:21.652447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.2.81.67:43405)
0924 12:48:21.652660 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 12:48:21.652665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:48:21.652692 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:48:21.652886 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:48:21.652891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:48:21.652966 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:48:21.653143 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:48:21.655096 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.655898 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.657539 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:48:21.657955 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:48:21.658082 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:48:21.663905 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:48:21.663923 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:48:21.663941 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:48:21.663986 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:48:21.667919 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.669089 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.669098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.669101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.669272 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:48:21.669698 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:48:21.669704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:48:21.669707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:48:21.670169 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:48:21.670496 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 12:48:21.670957 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 12:48:21.671211 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 12:48:21.672649 14198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:21.764326 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 14075
I20260924 12:48:21.777519 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 13681
I20260924 12:48:21.778862 13743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:48:21.890640 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 13681
I20260924 12:48:21.908753 2373 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 12:48:22.244629 2373 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 12:48:22.244850 2373 mini_ranger.cc:66] Stopping Ranger...
I20260924 12:48:22.581054 2373 mini_ranger.cc:68] Stopped Ranger
2026-09-24 12:48:22.581 UTC [13266] LOG: received smart shutdown request
2026-09-24 12:48:22.585 UTC [13266] LOG: background worker "logical replication launcher" (PID 13275) exited with exit code 1
2026-09-24 12:48:22.585 UTC [13270] LOG: shutting down
2026-09-24 12:48:22.798 UTC [13266] LOG: database system is shut down
2026-09-24T12:48:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53650 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:48:23 dist-test-slave-9tm1 krb5kdc[14262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:48:23 dist-test-slave-9tm1 krb5kdc[14262](info): set up 2 sockets
Sep 24 12:48:23 dist-test-slave-9tm1 krb5kdc[14262](info): commencing operation
krb5kdc: starting...
W20260924 12:48:23.961395 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:48:24 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:48:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/rangeradmin_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/rangeradmin_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/rangerlookup_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/rangerlookup_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
I20260924 12:48:24.287405 2373 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYQebFB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/postgres -l logfile start
2026-09-24 12:48:27.200 UTC [14292] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 12:48:27.200 UTC [14292] LOG: listening on IPv4 address "127.2.81.84", port 51573
2026-09-24 12:48:27.205 UTC [14292] LOG: listening on Unix socket "/tmp/.s.PGSQL.51573"
2026-09-24 12:48:27.228 UTC [14295] LOG: database system was shut down at 2026-09-24 12:48:25 UTC
2026-09-24 12:48:27.232 UTC [14292] LOG: database system is ready to accept connections
I20260924 12:48:28.128815 2373 mini_postgres.cc:96] Postgres bound to 51573
2026-09-24 12:48:28.145 UTC [14303] FATAL: database "slave" does not exist
127.2.81.84:51573 - accepting connections
I20260924 12:48:28.146149 2373 mini_ranger.cc:161] Starting Ranger...
I20260924 12:48:28.171408 2373 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 12:48:28.768661 2373 mini_ranger.cc:87] Created ranger Postgres database
I20260924 12:48:28.768764 2373 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-admin
2026-09-24 12:48:29,068 [I] DB FLAVOR :POSTGRES
2026-09-24 12:48:29,069 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 12:48:29,069 [I] Checking connection..
2026-09-24 12:48:29,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 12:48:29,319 [I] Checking connection passed.
2026-09-24 12:48:29,319 [I] --------- Verifying version history table ---------
2026-09-24 12:48:29,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 12:48:29,582 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 12:48:29,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 12:48:29,845 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 12:48:29,846 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 12:48:29,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 12:48:30.098 UTC [14367] WARNING: there is no transaction in progress
2026-09-24 12:48:30,105 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 12:48:30,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 12:48:30,372 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 12:48:30,373 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 12:48:30,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 12:48:30,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 12:48:30,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tm1.c.gcp-upstream.internal','N') ;"
2026-09-24 12:48:31,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 12:48:31,405 [I] Table x_portal_user does not exist in database ranger
2026-09-24 12:48:31,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 12:48:31,658 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 12:48:31,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 12:48:31,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 12:48:32,177 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 12:48:32,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 12:48:33.366 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.386 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.409 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.432 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.452 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.478 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.555 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.563 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.579 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.590 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.601 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.618 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.629 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.638 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.647 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:33.657 UTC [14475] WARNING: there is no transaction in progress
2026-09-24 12:48:34,503 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 12:48:34,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 12:48:34,758 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 12:48:34,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 12:48:35,025 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 12:48:35,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 12:48:35,271 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 12:48:35,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 12:48:35,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 12:48:35,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tm1.c.gcp-upstream.internal';"
2026-09-24 12:48:36,036 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 12:48:36,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 12:48:36,293 [I] DB_PATCHES have already been applied
I20260924 12:48:36.299470 2373 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYQebFB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 12:48:36.299569 2373 mini_ranger.cc:193] Using host: 127.2.81.84
I20260924 12:48:36.313134 2373 mini_ranger.cc:227] Ranger admin URL: http://127.2.81.84:38561
Sep 24, 2026 12:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:48:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:48:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 12:48:37.391271 2373 mini_ranger.cc:160] Time spent starting Ranger: real 9.245s user 0.000s sys 0.040s
Sep 24, 2026 12:48:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:48:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 12:48:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:48:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:49:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 12:49:00.885187 2373 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.81.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.81.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/rangerkms_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.81.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/rangerkms_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.81.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.81.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.81.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/HTTP_127.2.81.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 12:49:01 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254141, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 12:49:01.066811 2373 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 12:49:01.089953 2373 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 12:49:02.692864 2373 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 12:49:02.693056 2373 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms
I20260924 12:49:02.693104 2373 mini_ranger_kms.cc:227] Using postgres at 127.2.81.84:51573
2026-09-24 12:49:03,079 [I] DB FLAVOR :POSTGRES
2026-09-24 12:49:03,079 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 12:49:03,080 [I] Checking connection
2026-09-24 12:49:03,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 12:49:03,399 [I] connection success
2026-09-24 12:49:03,399 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 12:49:03,399 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 12:49:03,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 12:49:03,646 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 12:49:03,647 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 12:49:03,647 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 12:49:03,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.81.84:51573/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 12:49:03,918 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 12:49:04.484426 2373 mini_ranger_kms.cc:310] Created kms service
I20260924 12:49:04.677280 2373 mini_ranger_kms.cc:326] Created kudu user
I20260924 12:49:04.734885 2373 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 12:49:04 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254141, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:05.195604 2373 mini_ranger_kms.cc:384] Added ranger policy
I20260924 12:49:05.195910 2373 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYQebFB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYQebFB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 12:49:05.195968 2373 mini_ranger_kms.cc:240] Using host: 127.2.81.84
I20260924 12:49:05.216516 2373 mini_ranger_kms.cc:276] Ranger KMS PID: 14656
I20260924 12:49:05.216650 2373 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.81.84:50563
Sep 24, 2026 12:49:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:49:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:49:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:49:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:49:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:49:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 12:49:07.337582 2373 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.271s user 0.071s sys 0.315s
I20260924 12:49:07.337854 2373 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 12:49:07.338042 2373 mini_ranger_kms.cc:401] 127.2.81.84:50563/kms/v1/keys
Sep 24 12:49:07 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.81.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 12:49:07 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.81.84@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
Sep 24 12:49:08 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.81.84@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
Sep 24 12:49:10 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:49:10.733659 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41607
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:41607
--ranger_config_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:11.071719 14706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:11.072049 14706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:11.072161 14706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:11.079716 14706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:11.079850 14706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:11.079898 14706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:11.079979 14706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:11.080049 14706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:11.090592 14706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.81.84:50563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:41607
--ranger_config_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:11.091945 14706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:11.093520 14706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:11.105705 14712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:11.105792 14714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:11.105728 14711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:11.106446 14706 server_base.cc:1034] running on GCE node
I20260924 12:49:11.107232 14706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:11.108748 14706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:11.110018 14706 hybrid_clock.cc:648] HybridClock initialized: now 1790254151109931 us; error 57 us; skew 500 ppm
I20260924 12:49:11.110384 14706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:11 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254151, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:11.119630 14706 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:11.121337 14706 webserver.cc:460] Webserver started at http://127.2.81.126:44495/ using document root <none> and password file <none>
I20260924 12:49:11.122005 14706 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:11.122114 14706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:11.122380 14706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:49:11 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254151, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:11.182583 14706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "33897a84947d45178761dc428b295f4d"
format_stamp: "Formatted at 2026-09-24 12:49:11 on dist-test-slave-9tm1"
server_key: "6c298b63cc75d77a73e671f7f3301041"
server_key_iv: "28cb10064fc96b78eed394b1d1bbac46"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:11.183533 14706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "33897a84947d45178761dc428b295f4d"
format_stamp: "Formatted at 2026-09-24 12:49:11 on dist-test-slave-9tm1"
server_key: "6c298b63cc75d77a73e671f7f3301041"
server_key_iv: "28cb10064fc96b78eed394b1d1bbac46"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:11.189954 14706 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260924 12:49:11.208803 14721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:11.210691 14706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 12:49:11.210932 14706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "33897a84947d45178761dc428b295f4d"
format_stamp: "Formatted at 2026-09-24 12:49:11 on dist-test-slave-9tm1"
server_key: "6c298b63cc75d77a73e671f7f3301041"
server_key_iv: "28cb10064fc96b78eed394b1d1bbac46"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:11.211174 14706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:11.228785 14706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:11.233258 14706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:11.233562 14706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:11.255610 14706 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:41607
I20260924 12:49:11.255635 14772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:41607 every 8 connection(s)
I20260924 12:49:11.257308 14706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:11.257697 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 14706
I20260924 12:49:11.257885 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:49:11.258658 2373 external_mini_cluster.cc:1428] Setting key 4603a149e65ffd5059cc5bddd91a3a6b
I20260924 12:49:11.264770 14773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:11.285712 14773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Bootstrap starting.
Sep 24 12:49:11 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:11.289330 14773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:11.290871 14773 log.cc:824] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:11.295074 14773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: No bootstrap required, opened a new log
I20260924 12:49:11.299212 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:11.261308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.2.81.126:41607)
0924 12:49:11.270350 (+ 9042us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:11.270373 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:11.271295 (+ 922us) server_negotiation.cc:407] Connection header received
0924 12:49:11.274991 (+ 3696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:11.275038 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:11.276094 (+ 1056us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:11.276721 (+ 627us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:11.278320 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:11.280478 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:11.282617 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:11.283851 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:11.284092 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:11.290301 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:11.290315 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:11.290338 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:11.290395 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:11.294569 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:11.295453 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:11.295460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:11.295469 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:11.295717 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:11.296147 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:11.296151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:11.296153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:11.296777 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:11.297099 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:11.297612 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 12:49:11.298054 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8599,"thread_start_us":8187,"threads_started":1}
I20260924 12:49:11.305466 14773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:11.305889 14773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:11.305964 14773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33897a84947d45178761dc428b295f4d, State: Initialized, Role: FOLLOWER
I20260924 12:49:11.306658 14773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:11.306941 14773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:11.307099 14773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:11.307286 14773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:11.309613 14773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:11.310205 14773 leader_election.cc:302] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33897a84947d45178761dc428b295f4d; no voters:
I20260924 12:49:11.311023 14773 leader_election.cc:288] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:49:11.311233 14778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:11.312059 14778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 LEADER]: Becoming Leader. State: Replica: 33897a84947d45178761dc428b295f4d, State: Running, Role: LEADER
I20260924 12:49:11.312604 14778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:11.313395 14773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:11.315745 14779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33897a84947d45178761dc428b295f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } } }
I20260924 12:49:11.315766 14780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33897a84947d45178761dc428b295f4d. Latest consensus state: current_term: 1 leader_uuid: "33897a84947d45178761dc428b295f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } } }
I20260924 12:49:11.316357 14780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:11.316357 14779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:11.330281 14785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:11.336524 14785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:11.337457 14773 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 12:49:11.357034 14785 catalog_manager.cc:1347] Generated new cluster ID: e728fb9d05d5401a9f6e1a2ebe97fe9b
I20260924 12:49:11.357187 14785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:11.388855 14785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:49:11.391098 14785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:11.400044 14785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Generated new TSK 0
I20260924 12:49:11.401302 14785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:49:12.228499 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41607
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:49:12 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254152, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 12:49:12.688704 14818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:12.689190 14818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:12.689355 14818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:12.701624 14818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:12.701792 14818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:12.702036 14818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:49:12.718333 14818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.81.84:50563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:12.720124 14818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:12.721976 14818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:12.748620 14829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:12.749253 14828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:12.755558 14818 server_base.cc:1034] running on GCE node
W20260924 12:49:12.756513 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:12.759735 14818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:12.761195 14818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:12.762473 14818 hybrid_clock.cc:648] HybridClock initialized: now 1790254152762413 us; error 60 us; skew 500 ppm
I20260924 12:49:12.762887 14818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:12 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254152, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:12.767164 14818 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:49:12.769459 14818 webserver.cc:460] Webserver started at http://127.2.81.65:40233/ using document root <none> and password file <none>
I20260924 12:49:12.770457 14818 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:12.770623 14818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:12.770952 14818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:49:12 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254152, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:12.808308 14818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "8d32cfe4f9fe49889a15a58cc19ae10f"
format_stamp: "Formatted at 2026-09-24 12:49:12 on dist-test-slave-9tm1"
server_key: "d45d9ab9e1bcde241085d846cffc8434"
server_key_iv: "627544c271e5ea34a9989f688e4ac78f"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:12.809747 14818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "8d32cfe4f9fe49889a15a58cc19ae10f"
format_stamp: "Formatted at 2026-09-24 12:49:12 on dist-test-slave-9tm1"
server_key: "d45d9ab9e1bcde241085d846cffc8434"
server_key_iv: "627544c271e5ea34a9989f688e4ac78f"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:12.831700 14818 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.013s sys 0.008s
I20260924 12:49:12.871131 14838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:12.873550 14818 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.001s sys 0.004s
I20260924 12:49:12.874015 14818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "8d32cfe4f9fe49889a15a58cc19ae10f"
format_stamp: "Formatted at 2026-09-24 12:49:12 on dist-test-slave-9tm1"
server_key: "d45d9ab9e1bcde241085d846cffc8434"
server_key_iv: "627544c271e5ea34a9989f688e4ac78f"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:12.874425 14818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:12.922782 14818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:12.929045 14818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:12.929728 14818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:12.931313 14818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:12.934150 14818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:12.934293 14818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 12:49:12.934458 14818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:12.934541 14818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:49:12 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254152, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
Sep 24 12:49:13 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254152, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:13.484679 14951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:43493 every 8 connection(s)
I20260924 12:49:13.485136 14818 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:43493
I20260924 12:49:13.487056 14818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:49:13.492688 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 14818
I20260924 12:49:13.492894 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:49:13.493680 2373 external_mini_cluster.cc:1428] Setting key fe77b093cb96f40e3aaff26ce5d6ae1e
Sep 24 12:49:13 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254152, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:13.550298 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:13.495765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:54533 (local address 127.2.81.126:41607)
0924 12:49:13.521816 (+ 26051us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:13.521822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:13.525088 (+ 3266us) server_negotiation.cc:407] Connection header received
0924 12:49:13.526435 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:13.526442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:13.526540 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:13.526616 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:13.529993 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:13.531329 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:13.534140 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:13.534815 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:13.534917 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:13.541183 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:13.541190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:13.541195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:13.541241 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:13.545219 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:13.546605 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:13.546613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:13.546616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:13.546852 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:13.547805 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:13.547811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:13.547814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:13.548221 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:13.548375 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:13.549564 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0924 12:49:13.549867 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25890,"thread_start_us":26058,"threads_started":1}
I20260924 12:49:13.555071 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:13.495072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41607 (local address 127.2.81.65:54533)
0924 12:49:13.522497 (+ 27425us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:13.522557 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:13.526092 (+ 3535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:13.527212 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:13.527227 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:13.527885 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:13.529662 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:13.529703 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:13.531660 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:13.531667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:13.533921 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:13.533951 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:13.535279 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:13.535286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:13.535576 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:13.536408 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:13.536440 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:13.540790 (+ 4350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:13.545632 (+ 4842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:13.545655 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:13.545676 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:13.546416 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:13.547028 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:13.547033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:13.547037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:13.547604 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:13.548497 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:13.548506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:13.549043 (+ 537us) client_negotiation.cc:769] Sending connection context
0924 12:49:13.550879 (+ 1836us) client_negotiation.cc:240] Negotiation successful
0924 12:49:13.551837 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27057,"thread_start_us":29937,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:49:13.558126 14953 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41607
I20260924 12:49:13.558475 14953 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:13.559679 14953 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:13.562598 14738 ts_manager.cc:194] Registered new tserver with Master: 8d32cfe4f9fe49889a15a58cc19ae10f (127.2.81.65:43493)
I20260924 12:49:13.564697 14738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:54533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
W20260924 12:49:13.625114 2373 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.81.66: real 0.128s user 0.004s sys 0.013s
I20260924 12:49:13.626024 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41607
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:14.107465 14959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:14.107848 14959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:14.107991 14959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:14.115845 14959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:14.116010 14959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:14.116201 14959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:49:14.126833 14959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.81.84:50563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:14.128182 14959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:14.129526 14959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:14.144309 14971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:14.144380 14972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:14.145201 14974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:14.145581 14959 server_base.cc:1034] running on GCE node
I20260924 12:49:14.146241 14959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:14.147238 14959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:14.148470 14959 hybrid_clock.cc:648] HybridClock initialized: now 1790254154148382 us; error 65 us; skew 500 ppm
I20260924 12:49:14.148787 14959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254154, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:14.152091 14959 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:49:14.153931 14959 webserver.cc:460] Webserver started at http://127.2.81.66:44823/ using document root <none> and password file <none>
I20260924 12:49:14.154807 14959 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:14.154943 14959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:14.155287 14959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254154, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:14.174346 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "ab8d583c9c254d538343774c972e71a5"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "432c676d1bc47583b35b565094cf4f89"
server_key_iv: "23dccda75fc402ac1c4f88ad9ff39462"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:14.175674 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "ab8d583c9c254d538343774c972e71a5"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "432c676d1bc47583b35b565094cf4f89"
server_key_iv: "23dccda75fc402ac1c4f88ad9ff39462"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:14.185181 14959 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20260924 12:49:14.203833 14981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:14.206059 14959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:49:14.206292 14959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "ab8d583c9c254d538343774c972e71a5"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "432c676d1bc47583b35b565094cf4f89"
server_key_iv: "23dccda75fc402ac1c4f88ad9ff39462"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:14.206545 14959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:14.222436 14959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:14.226531 14959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:14.226827 14959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:14.227833 14959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:14.229274 14959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:14.229353 14959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:14.229475 14959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:14.229530 14959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:14.260155 14959 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:43791
I20260924 12:49:14.260212 15093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:43791 every 8 connection(s)
I20260924 12:49:14.261569 14959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:49:14.271003 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 14959
I20260924 12:49:14.271183 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:49:14.271634 2373 external_mini_cluster.cc:1428] Setting key 69064d4731ee5fa999717c7abee565a3
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254154, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:14.285758 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:14.264221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:34583 (local address 127.2.81.126:41607)
0924 12:49:14.264711 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:14.264716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:14.265697 (+ 981us) server_negotiation.cc:407] Connection header received
0924 12:49:14.267461 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:14.267466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:14.267551 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:14.267649 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:14.269234 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.270358 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.272306 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.272784 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.272922 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:14.276744 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:14.276751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:14.276757 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:14.276803 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:14.280808 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:14.281988 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:14.281996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:14.281999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:14.282552 (+ 553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:14.283296 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:14.283300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:14.283303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:14.283728 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:14.283883 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:14.284931 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0924 12:49:14.285412 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260924 12:49:14.286778 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:14.264677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41607 (local address 127.2.81.66:34583)
0924 12:49:14.265495 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:14.265560 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:14.267173 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:14.267914 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:14.267923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:14.268368 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:14.269027 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:14.269050 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.270554 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.270559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:14.272056 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:14.272080 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.273111 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.273119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:14.273378 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:14.274198 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:14.274252 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:14.276437 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:14.281139 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:14.281148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:14.281165 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:14.281784 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:14.282721 (+ 937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:14.282727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:14.282730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:14.283045 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:14.283918 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:14.283928 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:14.284399 (+ 471us) client_negotiation.cc:769] Sending connection context
0924 12:49:14.284722 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 12:49:14.285348 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":394,"threads_started":1}
I20260924 12:49:14.288530 15094 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41607
I20260924 12:49:14.288884 15094 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:14.289739 15094 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:14.291445 14738 ts_manager.cc:194] Registered new tserver with Master: ab8d583c9c254d538343774c972e71a5 (127.2.81.66:43791)
I20260924 12:49:14.292330 14738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:34583
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:49:14.329689 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41607
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:14.568866 14953 heartbeater.cc:503] Master 127.2.81.126:41607 was elected leader, sending a full tablet report...
W20260924 12:49:14.667884 15100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:14.668180 15100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:14.668299 15100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:14.675757 15100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:14.675884 15100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:14.676033 15100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:49:14.686374 15100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.81.84:50563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:14.687755 15100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:14.689103 15100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:14.704106 15106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:14.704262 15108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:14.704393 15100 server_base.cc:1034] running on GCE node
W20260924 12:49:14.704262 15105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:14.705420 15100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:14.706446 15100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:14.707683 15100 hybrid_clock.cc:648] HybridClock initialized: now 1790254154707630 us; error 43 us; skew 500 ppm
I20260924 12:49:14.708027 15100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254154, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:14.712253 15100 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:49:14.714926 15100 webserver.cc:460] Webserver started at http://127.2.81.67:35087/ using document root <none> and password file <none>
I20260924 12:49:14.715796 15100 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:14.715934 15100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:14.716281 15100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254154, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:14.732108 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "12d223bac76344d2986983ecfbd97390"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "6cacc0424b812a0db297fa116fbe9b5a"
server_key_iv: "475e77a4ed67279e14ce22cec8ceb429"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:14.733075 15100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "12d223bac76344d2986983ecfbd97390"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "6cacc0424b812a0db297fa116fbe9b5a"
server_key_iv: "475e77a4ed67279e14ce22cec8ceb429"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:14.739514 15100 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 12:49:14.760637 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:14.762456 15100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:49:14.762698 15100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "12d223bac76344d2986983ecfbd97390"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "6cacc0424b812a0db297fa116fbe9b5a"
server_key_iv: "475e77a4ed67279e14ce22cec8ceb429"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:14.762950 15100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:14.785743 15100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:14.791495 15100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:14.791921 15100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:14.793905 15100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:14.795475 15100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:14.795562 15100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:14.795696 15100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:14.795750 15100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:14.833334 15100 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:43463
I20260924 12:49:14.833385 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:43463 every 8 connection(s)
I20260924 12:49:14.834764 15100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:49:14.844430 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15100
I20260924 12:49:14.844595 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:49:14.845101 2373 external_mini_cluster.cc:1428] Setting key 4686ea6861ab002798bdd03b4594b170
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254154, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:14.861512 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:14.837721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:51511 (local address 127.2.81.126:41607)
0924 12:49:14.838356 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:14.838361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:14.839421 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 12:49:14.841352 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:14.841358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:14.841459 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:14.841561 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:14.843302 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.844069 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.846103 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.846832 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.846967 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:14.852249 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:14.852527 (+ 278us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:14.852537 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:14.852593 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:14.856842 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:14.858004 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:14.858010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:14.858012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:14.858263 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:14.859101 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:14.859106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:14.859108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:14.859463 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:14.859619 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:14.860727 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0924 12:49:14.861123 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":265,"threads_started":1}
I20260924 12:49:14.862538 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:14.838388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41607 (local address 127.2.81.67:51511)
0924 12:49:14.839227 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:14.839272 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:14.840999 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:14.841850 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:14.841859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:14.842329 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:14.843069 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:14.843089 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.844308 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.844313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:14.845883 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:14.845900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.847102 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.847119 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:14.847499 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:14.848393 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:14.848419 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:14.851554 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:14.857132 (+ 5578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:14.857144 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:14.857182 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:14.857747 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:14.858646 (+ 899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:14.858651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:14.858654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:14.858917 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:14.859686 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:14.859693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:14.860214 (+ 521us) client_negotiation.cc:769] Sending connection context
0924 12:49:14.860550 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 12:49:14.861183 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":357,"threads_started":1}
I20260924 12:49:14.864064 15228 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41607
I20260924 12:49:14.864387 15228 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:14.865235 15228 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:14.866920 14738 ts_manager.cc:194] Registered new tserver with Master: 12d223bac76344d2986983ecfbd97390 (127.2.81.67:43463)
I20260924 12:49:14.867782 14738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:51511
I20260924 12:49:14.875669 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:49:14.877458 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 14818
Sep 24 12:49:14 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:49:14.898689 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:14.878526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.2.81.65:43493)
0924 12:49:14.879156 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:14.879164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:14.879191 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:49:14.879250 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:14.879257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:14.879466 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:14.879604 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:14.881020 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.882370 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.884223 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:14.884863 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:14.885034 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:14.890861 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:14.890883 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:14.890900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:14.890949 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:14.895402 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:14.896361 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:14.896369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:14.896372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:14.896586 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:14.897053 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:14.897059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:14.897062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:14.897493 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:14.897770 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:14.898089 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 12:49:14.898306 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":343,"threads_started":1}
I20260924 12:49:14.900166 14947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:14.984771 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 14818
I20260924 12:49:14.997757 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 14959
Sep 24 12:49:15 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:49:15.019335 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:14.998668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.2.81.66:43791)
0924 12:49:14.999266 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:14.999275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:14.999301 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:49:14.999371 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:14.999378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:14.999608 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:14.999745 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:15.001130 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:15.002197 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:15.003681 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:15.004315 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:15.004441 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:15.012954 (+ 8513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:15.013013 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:15.013034 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:15.013093 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:15.016204 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:15.016956 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:15.016963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:15.016966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:15.017141 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:15.017582 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:15.017586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:15.017588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:15.017952 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:15.018279 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:15.018691 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 12:49:15.018905 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":269,"threads_started":1}
I20260924 12:49:15.020625 15089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:15.103221 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 14959
I20260924 12:49:15.115041 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15100
Sep 24 12:49:15 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:49:15.132357 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:15.115987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.2.81.67:43463)
0924 12:49:15.116578 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:15.116585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:15.116614 (+ 29us) server_negotiation.cc:407] Connection header received
0924 12:49:15.116674 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:15.116681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:15.116894 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:15.117099 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:15.118209 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:15.119316 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:15.120593 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:15.121249 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:15.121357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:15.126163 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:15.126183 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:15.126198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:15.126245 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:15.129506 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:15.130312 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:15.130320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:15.130322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:15.130470 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:15.130859 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:15.130862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:15.130864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:15.131245 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:15.131477 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:15.131764 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 12:49:15.131963 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":302,"threads_started":1}
I20260924 12:49:15.133785 15223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:15.217029 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15100
I20260924 12:49:15.229352 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 14706
I20260924 12:49:15.230639 14768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:15.341682 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 14706
I20260924 12:49:15.360291 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41607
--webserver_interface=127.2.81.126
--webserver_port=44495
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:41607
--ranger_config_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:15.694173 15240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:15.694501 15240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:15.694605 15240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:15.702037 15240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:15.702171 15240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:15.702234 15240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:15.702314 15240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:15.702376 15240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:15.712900 15240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:41607
--ranger_config_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=44495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:15.714257 15240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:15.715547 15240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:15.727685 15248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:15.727768 15246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:15.728204 15245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:15.728672 15240 server_base.cc:1034] running on GCE node
I20260924 12:49:15.729507 15240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:15.731042 15240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:15.732266 15240 hybrid_clock.cc:648] HybridClock initialized: now 1790254155732219 us; error 43 us; skew 500 ppm
I20260924 12:49:15.732645 15240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:15 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:15.735787 15240 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:15.737389 15240 webserver.cc:460] Webserver started at http://127.2.81.126:44495/ using document root <none> and password file <none>
I20260924 12:49:15.738049 15240 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:15.738160 15240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 12:49:15 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254155, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:15.808085 15240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:49:15.811568 15256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:15.813092 15240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:49:15.813273 15240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "33897a84947d45178761dc428b295f4d"
format_stamp: "Formatted at 2026-09-24 12:49:11 on dist-test-slave-9tm1"
server_key: "6c298b63cc75d77a73e671f7f3301041"
server_key_iv: "28cb10064fc96b78eed394b1d1bbac46"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:15.814232 15240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:15.846266 15240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:15.850449 15240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:15.850746 15240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:15.868759 15240 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:41607
I20260924 12:49:15.868778 15307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:41607 every 8 connection(s)
I20260924 12:49:15.870246 15240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:15.873883 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15240
I20260924 12:49:15.874668 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:43493
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:41607
--webserver_port=40233
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:15.877488 15308 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:49:15.881198 15308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Bootstrap starting.
I20260924 12:49:15.900789 15308 log.cc:824] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:15.909750 15308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:49:15.910254 15308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Bootstrap complete.
I20260924 12:49:15.919232 15308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:15.920269 15308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33897a84947d45178761dc428b295f4d, State: Initialized, Role: FOLLOWER
I20260924 12:49:15.920872 15308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:15.921142 15308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:15.921411 15308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:15.921631 15308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:49:15.925818 15308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:15.926357 15308 leader_election.cc:302] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33897a84947d45178761dc428b295f4d; no voters:
I20260924 12:49:15.926962 15308 leader_election.cc:288] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 12:49:15.927106 15312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:49:15.928169 15312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [term 2 LEADER]: Becoming Leader. State: Replica: 33897a84947d45178761dc428b295f4d, State: Running, Role: LEADER
I20260924 12:49:15.928786 15312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } }
I20260924 12:49:15.929147 15308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:15.931320 15313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "33897a84947d45178761dc428b295f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } } }
I20260924 12:49:15.931774 15313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:15.935536 15314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33897a84947d45178761dc428b295f4d. Latest consensus state: current_term: 2 leader_uuid: "33897a84947d45178761dc428b295f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33897a84947d45178761dc428b295f4d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 41607 } } }
I20260924 12:49:15.935973 15314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:15.937769 15318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:15.944135 15318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:15.945021 15308 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 12:49:15.945901 15318 catalog_manager.cc:1259] Loaded cluster ID: e728fb9d05d5401a9f6e1a2ebe97fe9b
I20260924 12:49:15.946077 15318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:15.950227 15318 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:15.951848 15318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Loaded TSK: 0
I20260924 12:49:15.953620 15318 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 12:49:16.325122 15310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:16.325639 15310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:16.325858 15310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:16.337990 15310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:16.338284 15310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:16.338533 15310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:49:16.351413 15310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:43493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=40233
--tserver_master_addrs=127.2.81.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:16.353770 15310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:16.356719 15310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:16.390753 15345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:16.392853 15346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:16.399046 15348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:16.399664 15310 server_base.cc:1034] running on GCE node
I20260924 12:49:16.400920 15310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:16.409490 15310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:16.413166 15310 hybrid_clock.cc:648] HybridClock initialized: now 1790254156413060 us; error 92 us; skew 500 ppm
I20260924 12:49:16.413676 15310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:16 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:16.425273 15310 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:49:16.430750 15310 webserver.cc:460] Webserver started at http://127.2.81.65:40233/ using document root <none> and password file <none>
I20260924 12:49:16.431860 15310 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:16.432062 15310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 12:49:16 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:16.475389 15310 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.007s sys 0.000s
I20260924 12:49:16.489754 15356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:16.498042 15310 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.006s sys 0.004s
I20260924 12:49:16.498301 15310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "8d32cfe4f9fe49889a15a58cc19ae10f"
format_stamp: "Formatted at 2026-09-24 12:49:12 on dist-test-slave-9tm1"
server_key: "d45d9ab9e1bcde241085d846cffc8434"
server_key_iv: "627544c271e5ea34a9989f688e4ac78f"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:16.499830 15310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:16.527127 15310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:16.533842 15310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:16.534189 15310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:16.535552 15310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:16.537549 15310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:16.537708 15310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 12:49:16.537858 15310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:16.537945 15310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:16.602741 15310 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:43493
I20260924 12:49:16.602802 15469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:43493 every 8 connection(s)
I20260924 12:49:16.604830 15310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:49:16.609835 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15310
I20260924 12:49:16.610647 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:43791
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:41607
--webserver_port=44823
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:49:16 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:16.640332 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:16.609116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:43019 (local address 127.2.81.126:41607)
0924 12:49:16.610863 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:16.610888 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:16.613136 (+ 2248us) server_negotiation.cc:407] Connection header received
0924 12:49:16.614578 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:16.614598 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:16.615179 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:16.615987 (+ 808us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:16.618234 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:16.619727 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:16.622240 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:16.623360 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:16.623573 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:16.628119 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:16.628130 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:16.628154 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:16.628218 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:16.632533 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:16.633807 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:16.633818 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:16.633827 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:16.634078 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:16.634699 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:16.634704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:16.634708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:16.635454 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:16.635667 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:16.637776 (+ 2109us) server_negotiation.cc:299] Negotiation successful
0924 12:49:16.638318 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":1146,"threads_started":1}
I20260924 12:49:16.641978 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:16.609806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41607 (local address 127.2.81.65:43019)
0924 12:49:16.611486 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:16.611541 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:16.613842 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:16.616428 (+ 2586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:16.616440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:16.617125 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:16.617962 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:16.617992 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:16.619951 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:16.619957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:16.621995 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:16.622018 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:16.623697 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:16.623706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:16.624001 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:16.624857 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:16.624899 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:16.627705 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:16.632779 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:16.632790 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:16.632808 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:16.633556 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:16.634244 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:16.634250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:16.634253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:16.634521 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:16.635753 (+ 1232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:16.635761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:16.636196 (+ 435us) client_negotiation.cc:769] Sending connection context
0924 12:49:16.636538 (+ 342us) client_negotiation.cc:240] Negotiation successful
0924 12:49:16.637200 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":1152,"threads_started":1}
I20260924 12:49:16.647647 15470 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41607
I20260924 12:49:16.648026 15470 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:16.649382 15470 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
W20260924 12:49:16.654542 15470 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:41607 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 12:49:16.655010 15470 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:16.655440 15470 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:16.657235 15470 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:16.657684 15470 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
W20260924 12:49:16.659178 15470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 12:49:16.885205 15481 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 12:49:17.002817 15474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:17.003121 15474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:17.003293 15474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:17.011332 15474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:17.011485 15474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:17.011687 15474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:49:17.022605 15474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:43791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=44823
--tserver_master_addrs=127.2.81.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:17.024020 15474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:17.025388 15474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:17.041275 15486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:17.042111 15487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:17.042644 15489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:17.044445 15474 server_base.cc:1034] running on GCE node
I20260924 12:49:17.045308 15474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:17.046643 15474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:17.047919 15474 hybrid_clock.cc:648] HybridClock initialized: now 1790254157047852 us; error 59 us; skew 500 ppm
I20260924 12:49:17.048300 15474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:17.052507 15474 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:49:17.054647 15474 webserver.cc:460] Webserver started at http://127.2.81.66:44823/ using document root <none> and password file <none>
I20260924 12:49:17.055698 15474 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:17.055868 15474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:17.086416 15474 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.009s sys 0.001s
I20260924 12:49:17.090369 15499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:17.092224 15474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:49:17.092432 15474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "ab8d583c9c254d538343774c972e71a5"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "432c676d1bc47583b35b565094cf4f89"
server_key_iv: "23dccda75fc402ac1c4f88ad9ff39462"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:17.093941 15474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:17.127240 15474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:17.133865 15474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:17.134208 15474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:17.135521 15474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:17.137441 15474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:17.137574 15474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:17.137739 15474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:17.137820 15474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:17.254844 15474 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:43791
I20260924 12:49:17.254925 15613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:43791 every 8 connection(s)
I20260924 12:49:17.256829 15474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:49:17.265493 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15474
I20260924 12:49:17.266286 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.81.84:50563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:43463
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:41607
--webserver_port=35087
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:17.289878 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:17.261259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:53529 (local address 127.2.81.126:41607)
0924 12:49:17.262327 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:17.262335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:17.269031 (+ 6696us) server_negotiation.cc:407] Connection header received
0924 12:49:17.269119 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:17.269124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:17.269234 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:17.269342 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:17.271801 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:17.272828 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:17.276128 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:17.276809 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:17.276919 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:17.281743 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:17.281750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:17.281754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:17.281800 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:17.285274 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:17.286468 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:17.286475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:17.286480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:17.286673 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:17.287289 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:17.287295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:17.287298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:17.287702 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:17.287838 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:17.289050 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0924 12:49:17.289480 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":2064,"threads_started":1}
I20260924 12:49:17.293557 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:17.262744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41607 (local address 127.2.81.66:53529)
0924 12:49:17.263591 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:17.263638 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:17.268474 (+ 4836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:17.269887 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:17.269900 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:17.270610 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:17.271519 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:17.271552 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:17.273097 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:17.273103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:17.275891 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:17.275915 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:17.277317 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:17.277324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:17.277653 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:17.278510 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:17.278542 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:17.281394 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:17.285540 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:17.285551 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:17.285572 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:17.286277 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:17.286849 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:17.286854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:17.286858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:17.287143 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:17.287956 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:17.287966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:17.288480 (+ 514us) client_negotiation.cc:769] Sending connection context
0924 12:49:17.290709 (+ 2229us) client_negotiation.cc:240] Negotiation successful
0924 12:49:17.291742 (+ 1033us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":587,"threads_started":1}
I20260924 12:49:17.296406 15614 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41607
I20260924 12:49:17.296792 15614 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:17.297760 15614 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:17.300487 15273 ts_manager.cc:194] Registered new tserver with Master: ab8d583c9c254d538343774c972e71a5 (127.2.81.66:43791)
I20260924 12:49:17.302834 15273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:53529
I20260924 12:49:17.659350 15470 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:17.659823 15470 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:17.661300 15273 ts_manager.cc:194] Registered new tserver with Master: 8d32cfe4f9fe49889a15a58cc19ae10f (127.2.81.65:43493)
I20260924 12:49:17.662541 15273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:43019
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
W20260924 12:49:17.761689 15618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:17.762183 15618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:17.762375 15618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:17.774309 15618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:17.774556 15618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:17.774794 15618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:49:17.790977 15618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.81.84:50563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:43463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=35087
--tserver_master_addrs=127.2.81.126:41607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:17.792948 15618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:17.795193 15618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:17.832167 15624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:17.838160 15625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:17.843722 15628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:17.844332 15618 server_base.cc:1034] running on GCE node
I20260924 12:49:17.845511 15618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:17.851985 15618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:17.853564 15618 hybrid_clock.cc:648] HybridClock initialized: now 1790254157853495 us; error 80 us; skew 500 ppm
I20260924 12:49:17.854077 15618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:17.858804 15618 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:49:17.879825 15618 webserver.cc:460] Webserver started at http://127.2.81.67:35087/ using document root <none> and password file <none>
I20260924 12:49:17.880960 15618 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:17.881243 15618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 12:49:17 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:17.939908 15618 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.000s
I20260924 12:49:17.955014 15635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:17.957569 15618 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.007s
I20260924 12:49:17.957873 15618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "12d223bac76344d2986983ecfbd97390"
format_stamp: "Formatted at 2026-09-24 12:49:14 on dist-test-slave-9tm1"
server_key: "6cacc0424b812a0db297fa116fbe9b5a"
server_key_iv: "475e77a4ed67279e14ce22cec8ceb429"
server_key_version: "kuduclusterkey@0"
I20260924 12:49:17.959424 15618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:18.009766 15618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:18.016039 15618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:18.016409 15618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:18.023910 15618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:18.032248 15618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:18.032474 15618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:18.032632 15618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:18.032732 15618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 12:49:18 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for HTTP/127.2.81.84@KRBTEST.COM
I20260924 12:49:18.308241 15614 heartbeater.cc:503] Master 127.2.81.126:41607 was elected leader, sending a full tablet report...
I20260924 12:49:18.558498 15618 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:43463
I20260924 12:49:18.558528 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:43463 every 8 connection(s)
I20260924 12:49:18.560060 15618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:49:18.570209 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15618
Sep 24 12:49:18 dist-test-slave-9tm1 krb5kdc[14262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254157, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:18.589860 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.563048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:48659 (local address 127.2.81.126:41607)
0924 12:49:18.568425 (+ 5377us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.568431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.568456 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:49:18.570019 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.570025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.570132 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.570231 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:18.572404 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.573732 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.576613 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.577649 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.577821 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.582347 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:18.582353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:18.582358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:18.582399 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:18.585830 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.586795 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.586801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.586804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.586975 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.587499 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.587503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.587506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.587775 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:18.587927 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.588953 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.589344 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5254,"thread_start_us":5133,"threads_started":1}
I20260924 12:49:18.590886 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.563582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:41607 (local address 127.2.81.67:48659)
0924 12:49:18.567538 (+ 3956us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:18.567604 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:18.569670 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:18.570556 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:18.570569 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:18.571179 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:18.572113 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.572142 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.573979 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.573987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.576328 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.576354 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.578284 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.578293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.578690 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.579576 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:18.579616 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:18.582038 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:18.586005 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:18.586017 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:18.586044 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:18.586667 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:18.587105 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:18.587110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:18.587114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:18.587361 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:18.587942 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:18.587951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:18.588394 (+ 443us) client_negotiation.cc:769] Sending connection context
0924 12:49:18.588761 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 12:49:18.589329 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3561,"thread_start_us":3311,"threads_started":1}
I20260924 12:49:18.594920 15756 heartbeater.cc:347] Connected to a master server at 127.2.81.126:41607
I20260924 12:49:18.595263 15756 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:18.596158 15756 heartbeater.cc:511] Master 127.2.81.126:41607 requested a full tablet report, sending...
I20260924 12:49:18.597898 15273 ts_manager.cc:194] Registered new tserver with Master: 12d223bac76344d2986983ecfbd97390 (127.2.81.67:43463)
I20260924 12:49:18.599002 15273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:48659
I20260924 12:49:18.603582 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.571223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.2.81.126:41607)
0924 12:49:18.581439 (+ 10216us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.581444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.581460 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:49:18.581509 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.581513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.581582 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.581674 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:18.583345 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.584609 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.586464 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.586981 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.587099 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.595598 (+ 8499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:18.595605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:18.595610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:18.595657 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:18.599661 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.601371 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.601377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.601381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.601550 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.602070 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.602075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.602078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.602417 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:18.602736 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.602979 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.603218 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10057,"thread_start_us":9920,"threads_started":1}
I20260924 12:49:18.606670 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:49:18.624658 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.610219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.2.81.126:41607)
0924 12:49:18.610429 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.610435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.610821 (+ 386us) server_negotiation.cc:407] Connection header received
0924 12:49:18.610976 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.610981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.611067 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.611165 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:18.612610 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.613747 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.615200 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.615894 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.616026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.618558 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:18.618564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:18.618569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:18.618620 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:18.621877 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.622637 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.622643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.622646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.622824 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.623168 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.623173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.623177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.623519 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:18.623779 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.624065 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.624311 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:49:18.630877 15273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:49:18.643244 15273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:49:18.666955 15470 heartbeater.cc:503] Master 127.2.81.126:41607 was elected leader, sending a full tablet report...
I20260924 12:49:18.698355 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.661525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43463 (local address 127.0.0.1:47492)
0924 12:49:18.687033 (+ 25508us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:18.687084 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:18.687310 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:18.689542 (+ 2232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:18.689550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:18.689583 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:18.690039 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.690060 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.692548 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.692553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.694969 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.694986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.697307 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.697315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.697530 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.697581 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 12:49:18.697729 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 12:49:18.697936 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25208,"thread_start_us":25970,"threads_started":1}
I20260924 12:49:18.699172 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.662241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.2.81.67:43463)
0924 12:49:18.688929 (+ 26688us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.688935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.688962 (+ 27us) server_negotiation.cc:407] Connection header received
0924 12:49:18.689074 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.689081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.689303 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.689513 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:18.690319 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.692384 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.695184 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.696686 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.696810 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.697077 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.698645 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.698833 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26539,"thread_start_us":26773,"threads_started":1}
I20260924 12:49:18.699846 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.661620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:43791 (local address 127.0.0.1:47532)
0924 12:49:18.687785 (+ 26165us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:18.687804 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:18.687921 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:18.691822 (+ 3901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:18.691827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:18.691853 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:18.692161 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.692171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.694338 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.694344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.696488 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.696504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.698819 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.698826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.698995 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.699030 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:49:18.699108 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 12:49:18.699509 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26037,"mutex_wait_us":2,"thread_start_us":26359,"threads_started":1}
I20260924 12:49:18.701799 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.662060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.2.81.66:43791)
0924 12:49:18.690941 (+ 28881us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.690951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.690981 (+ 30us) server_negotiation.cc:407] Connection header received
0924 12:49:18.691087 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.691095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.691366 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.691578 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:18.692356 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.694153 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.696696 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.698184 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.698306 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.701252 (+ 2946us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.701332 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.701498 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28701,"thread_start_us":30943,"threads_started":1}
I20260924 12:49:18.703234 15684 tablet_service.cc:1467] Processing CreateTablet for tablet cea40e7201e34336bfd69c3706ef8aa7 (DEFAULT_TABLE table=test-table [id=fd2dfd76dbfd4a23b71529dfe958963c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:18.704428 15684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea40e7201e34336bfd69c3706ef8aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:18.705364 15549 tablet_service.cc:1467] Processing CreateTablet for tablet cea40e7201e34336bfd69c3706ef8aa7 (DEFAULT_TABLE table=test-table [id=fd2dfd76dbfd4a23b71529dfe958963c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:18.706238 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.661621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55284)
0924 12:49:18.688116 (+ 26495us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:18.688177 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:18.688315 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:18.689868 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:18.689874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:18.689896 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:18.690361 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.690372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.695497 (+ 5125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.695502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.699270 (+ 3768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.699285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.701564 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.701571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.701756 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.701791 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:49:18.705592 (+ 3801us) client_negotiation.cc:240] Negotiation successful
0924 12:49:18.705754 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26332,"mutex_wait_us":156,"thread_start_us":24796,"threads_started":1}
I20260924 12:49:18.706800 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea40e7201e34336bfd69c3706ef8aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:18.707517 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.662455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.2.81.65:43493)
0924 12:49:18.688838 (+ 26383us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.688848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.688879 (+ 31us) server_negotiation.cc:407] Connection header received
0924 12:49:18.689010 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.689020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.689313 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.689517 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:18.692203 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.695318 (+ 3115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.699472 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.701089 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.701217 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.701425 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.706991 (+ 5566us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.707187 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26199,"thread_start_us":27259,"threads_started":1}
I20260924 12:49:18.711561 15405 tablet_service.cc:1467] Processing CreateTablet for tablet cea40e7201e34336bfd69c3706ef8aa7 (DEFAULT_TABLE table=test-table [id=fd2dfd76dbfd4a23b71529dfe958963c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:18.713303 15405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea40e7201e34336bfd69c3706ef8aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:18.720460 15777 tablet_bootstrap.cc:492] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Bootstrap starting.
I20260924 12:49:18.724291 15777 tablet_bootstrap.cc:654] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:18.724555 15778 tablet_bootstrap.cc:492] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Bootstrap starting.
I20260924 12:49:18.725597 15777 log.cc:824] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:18.728133 15778 tablet_bootstrap.cc:654] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:18.729060 15777 tablet_bootstrap.cc:492] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: No bootstrap required, opened a new log
I20260924 12:49:18.729353 15777 ts_tablet_manager.cc:1397] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:49:18.729372 15780 tablet_bootstrap.cc:492] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Bootstrap starting.
I20260924 12:49:18.729460 15778 log.cc:824] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:18.732121 15778 tablet_bootstrap.cc:492] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: No bootstrap required, opened a new log
I20260924 12:49:18.732368 15778 ts_tablet_manager.cc:1397] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 12:49:18.732992 15780 tablet_bootstrap.cc:654] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:18.734228 15780 log.cc:824] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:18.738751 15780 tablet_bootstrap.cc:492] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: No bootstrap required, opened a new log
I20260924 12:49:18.739135 15780 ts_tablet_manager.cc:1397] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260924 12:49:18.739874 15777 raft_consensus.cc:348] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.740361 15777 raft_consensus.cc:374] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:18.740479 15777 raft_consensus.cc:729] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d223bac76344d2986983ecfbd97390, State: Initialized, Role: FOLLOWER
I20260924 12:49:18.741222 15777 consensus_queue.cc:260] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.741268 15778 raft_consensus.cc:348] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.741742 15778 raft_consensus.cc:374] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:18.741839 15778 raft_consensus.cc:729] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8d583c9c254d538343774c972e71a5, State: Initialized, Role: FOLLOWER
I20260924 12:49:18.742376 15778 consensus_queue.cc:260] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.743903 15778 ts_tablet_manager.cc:1428] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 12:49:18.745548 15756 heartbeater.cc:503] Master 127.2.81.126:41607 was elected leader, sending a full tablet report...
I20260924 12:49:18.746220 15777 ts_tablet_manager.cc:1428] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260924 12:49:18.750730 15780 raft_consensus.cc:348] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.751209 15780 raft_consensus.cc:374] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:18.751317 15780 raft_consensus.cc:729] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d32cfe4f9fe49889a15a58cc19ae10f, State: Initialized, Role: FOLLOWER
I20260924 12:49:18.751935 15780 consensus_queue.cc:260] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.753648 15780 ts_tablet_manager.cc:1428] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260924 12:49:18.767913 15615 tablet.cc:2369] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:18.803332 15785 raft_consensus.cc:482] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:49:18.803637 15785 raft_consensus.cc:504] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.805534 15785 leader_election.cc:288] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12d223bac76344d2986983ecfbd97390 (127.2.81.67:43463), ab8d583c9c254d538343774c972e71a5 (127.2.81.66:43791)
I20260924 12:49:18.811952 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.806067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:60841 (local address 127.2.81.67:43463)
0924 12:49:18.806451 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.806457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.806750 (+ 293us) server_negotiation.cc:407] Connection header received
0924 12:49:18.806969 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.806974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.807057 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.807159 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:18.807746 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.808484 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.810315 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.811079 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.811223 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.811324 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.811597 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.811723 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260924 12:49:18.812098 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.805868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43463 (local address 127.2.81.65:60841)
0924 12:49:18.806626 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:18.806647 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:18.806830 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:18.807199 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:18.807202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:18.807223 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:18.807567 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.807579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.808614 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.808619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.810163 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.810175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.811295 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.811300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.811440 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.811465 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:49:18.811556 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 12:49:18.811689 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":32,"thread_start_us":395,"threads_started":1}
I20260924 12:49:18.812206 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.806077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:45323 (local address 127.2.81.66:43791)
0924 12:49:18.806297 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.806304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.807015 (+ 711us) server_negotiation.cc:407] Connection header received
0924 12:49:18.807117 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.807123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.807205 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.807283 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:18.808031 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.808758 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.810497 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.811337 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.811465 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.811562 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.811842 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.811965 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 12:49:18.812357 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.805941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:43791 (local address 127.2.81.65:45323)
0924 12:49:18.806910 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:18.806930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:18.807026 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:18.807379 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:18.807384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:18.807411 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:18.807817 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.807828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.808903 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.808908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.810352 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:18.810364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.811557 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.811564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:18.811700 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.811724 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 12:49:18.811799 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 12:49:18.811924 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":34,"thread_start_us":664,"threads_started":1}
I20260924 12:49:18.813374 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea40e7201e34336bfd69c3706ef8aa7" candidate_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d223bac76344d2986983ecfbd97390" is_pre_election: true
I20260924 12:49:18.813374 15569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea40e7201e34336bfd69c3706ef8aa7" candidate_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8d583c9c254d538343774c972e71a5" is_pre_election: true
W20260924 12:49:18.813959 15757 tablet.cc:2369] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:18.813967 15569 raft_consensus.cc:2454] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d32cfe4f9fe49889a15a58cc19ae10f in term 0.
I20260924 12:49:18.813978 15704 raft_consensus.cc:2454] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d32cfe4f9fe49889a15a58cc19ae10f in term 0.
I20260924 12:49:18.814766 15358 leader_election.cc:302] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d32cfe4f9fe49889a15a58cc19ae10f, ab8d583c9c254d538343774c972e71a5; no voters:
I20260924 12:49:18.815172 15785 raft_consensus.cc:2781] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:49:18.815310 15785 raft_consensus.cc:482] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:49:18.815410 15785 raft_consensus.cc:3037] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:18.817644 15785 raft_consensus.cc:504] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.818270 15785 leader_election.cc:288] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [CANDIDATE]: Term 1 election: Requested vote from peers 12d223bac76344d2986983ecfbd97390 (127.2.81.67:43463), ab8d583c9c254d538343774c972e71a5 (127.2.81.66:43791)
I20260924 12:49:18.818766 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea40e7201e34336bfd69c3706ef8aa7" candidate_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d223bac76344d2986983ecfbd97390"
I20260924 12:49:18.818810 15569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea40e7201e34336bfd69c3706ef8aa7" candidate_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8d583c9c254d538343774c972e71a5"
I20260924 12:49:18.818954 15704 raft_consensus.cc:3037] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:18.819038 15569 raft_consensus.cc:3037] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:18.821139 15704 raft_consensus.cc:2454] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d32cfe4f9fe49889a15a58cc19ae10f in term 1.
I20260924 12:49:18.821372 15569 raft_consensus.cc:2454] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d32cfe4f9fe49889a15a58cc19ae10f in term 1.
I20260924 12:49:18.821650 15359 leader_election.cc:302] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d223bac76344d2986983ecfbd97390, 8d32cfe4f9fe49889a15a58cc19ae10f; no voters:
I20260924 12:49:18.821993 15785 raft_consensus.cc:2781] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:18.822505 15785 raft_consensus.cc:686] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 1 LEADER]: Becoming Leader. State: Replica: 8d32cfe4f9fe49889a15a58cc19ae10f, State: Running, Role: LEADER
I20260924 12:49:18.823014 15785 consensus_queue.cc:237] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } }
I20260924 12:49:18.827163 15272 catalog_manager.cc:5495] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d32cfe4f9fe49889a15a58cc19ae10f (127.2.81.65). New cstate: current_term: 1 leader_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d32cfe4f9fe49889a15a58cc19ae10f" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 43493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 } health_report { overall_health: UNKNOWN } } }
W20260924 12:49:18.865705 15471 tablet.cc:2369] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:18.868005 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.862213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.2.81.65:43493)
0924 12:49:18.862440 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.862445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.862464 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:49:18.862556 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.862560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.862677 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.862820 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:49:18.863219 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.863971 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.865383 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.866089 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.866350 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.866846 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:49:18.867373 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:49:18.867487 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.867675 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.867753 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:49:18.878297 15569 raft_consensus.cc:1261] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 1 FOLLOWER]: Refusing update from remote peer 8d32cfe4f9fe49889a15a58cc19ae10f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:18.878558 15704 raft_consensus.cc:1261] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 1 FOLLOWER]: Refusing update from remote peer 8d32cfe4f9fe49889a15a58cc19ae10f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:18.879518 15788 consensus_queue.cc:1035] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d223bac76344d2986983ecfbd97390" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:18.879978 15785 consensus_queue.cc:1035] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8d583c9c254d538343774c972e71a5" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:18.894011 15792 mvcc.cc:205] Tried to move back new op lower bound from 7332881034752114688 to 7332881034542514176. Current Snapshot: MvccSnapshot[applied={T|T < 7332881034752114688}]
I20260924 12:49:18.907236 15272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:18.907509 15272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:18.911181 15272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33897a84947d45178761dc428b295f4d: Sending DeleteTablet for 3 replicas of tablet cea40e7201e34336bfd69c3706ef8aa7
I20260924 12:49:18.912500 15684 tablet_service.cc:1514] Processing DeleteTablet for tablet cea40e7201e34336bfd69c3706ef8aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:18 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:47492
I20260924 12:49:18.912614 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet cea40e7201e34336bfd69c3706ef8aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:18 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:47532
I20260924 12:49:18.912775 15405 tablet_service.cc:1514] Processing DeleteTablet for tablet cea40e7201e34336bfd69c3706ef8aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:18 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:55284
I20260924 12:49:18.922539 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15310
I20260924 12:49:18.925089 15800 tablet_replica.cc:330] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: stopping tablet replica
I20260924 12:49:18.925794 15800 raft_consensus.cc:2229] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:18.926569 15800 raft_consensus.cc:2258] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:18.928786 15800 ts_tablet_manager.cc:1905] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:18.936208 15801 tablet_replica.cc:330] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: stopping tablet replica
I20260924 12:49:18.936702 15801 raft_consensus.cc:2229] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:18.937299 15801 raft_consensus.cc:2258] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:18.937779 15800 ts_tablet_manager.cc:1918] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:18.938347 15802 tablet_replica.cc:330] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: stopping tablet replica
I20260924 12:49:18.938400 15800 log.cc:1196] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/cea40e7201e34336bfd69c3706ef8aa7
I20260924 12:49:18.939260 15800 ts_tablet_manager.cc:1939] T cea40e7201e34336bfd69c3706ef8aa7 P 12d223bac76344d2986983ecfbd97390: Deleting consensus metadata
I20260924 12:49:18.940738 15801 ts_tablet_manager.cc:1905] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:18.941507 15259 catalog_manager.cc:4841] TS 12d223bac76344d2986983ecfbd97390 (127.2.81.67:43463): tablet cea40e7201e34336bfd69c3706ef8aa7 (table test-table [id=fd2dfd76dbfd4a23b71529dfe958963c]) successfully deleted
I20260924 12:49:18.942551 15802 raft_consensus.cc:2229] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:49:18.943305 15802 raft_consensus.cc:2258] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:18.945487 15802 ts_tablet_manager.cc:1905] T cea40e7201e34336bfd69c3706ef8aa7 P 8d32cfe4f9fe49889a15a58cc19ae10f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:18.946632 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:18.924326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.2.81.65:43493)
0924 12:49:18.924503 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:18.924508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:18.924530 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:49:18.924596 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:18.924600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:18.924682 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:18.924788 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:18.926730 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.928015 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.931272 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:18.932054 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:18.932188 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:18.936040 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:18.936062 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:18.936084 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:18.936138 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:18.941406 (+ 5268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.942519 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.942534 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.942537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.942843 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:18.943382 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:18.943388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:18.943392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:18.943984 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:18.944384 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:18.945908 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0924 12:49:18.946232 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 12:49:18.947741 15465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:18.948575 15801 ts_tablet_manager.cc:1918] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:18.948782 15801 log.cc:1196] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/cea40e7201e34336bfd69c3706ef8aa7
I20260924 12:49:18.949436 15801 ts_tablet_manager.cc:1939] T cea40e7201e34336bfd69c3706ef8aa7 P ab8d583c9c254d538343774c972e71a5: Deleting consensus metadata
I20260924 12:49:18.951668 15258 catalog_manager.cc:4841] TS ab8d583c9c254d538343774c972e71a5 (127.2.81.66:43791): tablet cea40e7201e34336bfd69c3706ef8aa7 (table test-table [id=fd2dfd76dbfd4a23b71529dfe958963c]) successfully deleted
I20260924 12:49:19.046761 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15310
W20260924 12:49:19.060962 15257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:43493 (error 108)
I20260924 12:49:19.061199 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15474
I20260924 12:49:19.062943 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.062427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55318)
0924 12:49:19.062635 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.062767 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 12:49:19.063215 15257 catalog_manager.cc:4584] TS 8d32cfe4f9fe49889a15a58cc19ae10f (127.2.81.65:43493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cea40e7201e34336bfd69c3706ef8aa7: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
I20260924 12:49:19.075207 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.061940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.2.81.66:43791)
0924 12:49:19.062101 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:19.062105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:19.062207 (+ 102us) server_negotiation.cc:407] Connection header received
0924 12:49:19.062371 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:19.062376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:19.062460 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:19.062565 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:19.064039 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:19.064804 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:19.066162 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:19.066568 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:19.066666 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:19.069101 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:19.069114 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:19.069129 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:19.069169 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:19.072025 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:19.072857 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:19.072865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:19.072868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:19.073294 (+ 426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:19.073700 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:19.073703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:19.073705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:19.074059 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:19.074319 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:19.074620 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 12:49:19.074800 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:49:19.075933 15609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:19.116864 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.116395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55328)
0924 12:49:19.116619 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.116721 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 12:49:19.117937 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.117559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55332)
0924 12:49:19.117719 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.117797 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 12:49:19.165825 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15474
I20260924 12:49:19.179549 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.178702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55348)
0924 12:49:19.179111 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.179309 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260924 12:49:19.180907 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.180470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55350)
0924 12:49:19.180651 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.180749 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 12:49:19.182175 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15618
I20260924 12:49:19.200189 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.183191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.2.81.67:43463)
0924 12:49:19.183357 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:19.183361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:19.183378 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:49:19.183432 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:19.183436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:19.183505 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:19.183599 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:19.185195 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:19.186008 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:19.187422 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:19.187843 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:19.187947 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:19.192411 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:19.192437 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:19.192459 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:19.192517 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:19.196633 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:19.197885 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:19.197894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:19.197897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:19.198090 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:19.198667 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:19.198672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:19.198675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:19.199088 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:19.199379 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:19.199685 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 12:49:19.199869 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:49:19.200851 15751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:19.261910 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.261414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55352)
0924 12:49:19.261635 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.261752 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 12:49:19.262940 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:19.262553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:43493 (local address 127.0.0.1:55368)
0924 12:49:19.262764 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:19.262833 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 12:49:19.292670 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15618
I20260924 12:49:19.305701 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15240
I20260924 12:49:19.306871 15303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:19.421846 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15240
I20260924 12:49:19.442461 2373 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 12:49:19.779420 2373 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 12:49:19.779626 2373 mini_ranger.cc:66] Stopping Ranger...
I20260924 12:49:20.126178 2373 mini_ranger.cc:68] Stopped Ranger
2026-09-24 12:49:20.126 UTC [14292] LOG: received smart shutdown request
2026-09-24 12:49:20.131 UTC [14292] LOG: background worker "logical replication launcher" (PID 14301) exited with exit code 1
2026-09-24 12:49:20.131 UTC [14296] LOG: shutting down
2026-09-24 12:49:20.370 UTC [14292] LOG: database system is shut down
2026-09-24T12:49:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57576 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:49:20 dist-test-slave-9tm1 krb5kdc[15816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:49:20 dist-test-slave-9tm1 krb5kdc[15816](info): set up 2 sockets
Sep 24 12:49:20 dist-test-slave-9tm1 krb5kdc[15816](info): commencing operation
krb5kdc: starting...
W20260924 12:49:21.533661 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:49:21 dist-test-slave-9tm1 krb5kdc[15816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:49:21.789837 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:34751
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:34751,127.2.81.125:33081,127.2.81.124:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:22.170405 15829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:22.170760 15829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:22.170892 15829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:22.179160 15829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:22.179320 15829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:22.179394 15829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:22.179474 15829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:22.179538 15829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:22.193738 15829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:34751,127.2.81.125:33081,127.2.81.124:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:22.195334 15829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:22.197640 15829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:22.215130 15835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:22.215101 15834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:22.215257 15837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:22.216401 15829 server_base.cc:1034] running on GCE node
I20260924 12:49:22.217384 15829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:22.219156 15829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:22.220459 15829 hybrid_clock.cc:648] HybridClock initialized: now 1790254162220397 us; error 61 us; skew 500 ppm
I20260924 12:49:22.220884 15829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:22 dist-test-slave-9tm1 krb5kdc[15816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:22.229558 15829 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:22.231422 15829 webserver.cc:460] Webserver started at http://127.2.81.126:42591/ using document root <none> and password file <none>
I20260924 12:49:22.232240 15829 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:22.232389 15829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:22.232712 15829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:22.236044 15829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e"
format_stamp: "Formatted at 2026-09-24 12:49:22 on dist-test-slave-9tm1"
server_key: "200526fb6c56210c386bab7c8f465e0d"
server_key_iv: "92af2f43ae102160449caeb09cd736cd"
server_key_version: "encryptionkey@0"
I20260924 12:49:22.236948 15829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e"
format_stamp: "Formatted at 2026-09-24 12:49:22 on dist-test-slave-9tm1"
server_key: "200526fb6c56210c386bab7c8f465e0d"
server_key_iv: "92af2f43ae102160449caeb09cd736cd"
server_key_version: "encryptionkey@0"
I20260924 12:49:22.243145 15829 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 12:49:22.247478 15844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:22.249140 15829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:49:22.249337 15829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e"
format_stamp: "Formatted at 2026-09-24 12:49:22 on dist-test-slave-9tm1"
server_key: "200526fb6c56210c386bab7c8f465e0d"
server_key_iv: "92af2f43ae102160449caeb09cd736cd"
server_key_version: "encryptionkey@0"
I20260924 12:49:22.249559 15829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:22.269414 15829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:22.274251 15829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:22.274564 15829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:22.296639 15829 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:34751
I20260924 12:49:22.296661 15895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:34751 every 8 connection(s)
I20260924 12:49:22.298106 15829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:22.302445 15896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:22.305579 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15829
I20260924 12:49:22.305837 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:49:22.306614 2373 external_mini_cluster.cc:1428] Setting key 0a2f0cd1467c0b2612418156a56c7427
I20260924 12:49:22.311801 15896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:49:22 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.125@KRBTEST.COM" created.
I20260924 12:49:22.339912 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.317258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:34751 (local address 127.0.0.1:36506)
0924 12:49:22.318799 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.318840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:22.320918 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:22.322049 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:22.322055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:22.322837 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:22.323213 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:22.323233 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.324923 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.324927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:22.326221 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:22.326236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.327432 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.327440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:22.327627 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:22.328461 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:22.328489 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:22.331095 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:22.335006 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:22.335019 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:22.335034 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:22.335457 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:22.335914 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:22.335920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:22.335924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:22.336090 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:22.336926 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:22.336934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:22.337190 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 12:49:22.337416 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 12:49:22.337915 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"spinlock_wait_cycles":48000,"thread_start_us":1071,"threads_started":1}
I20260924 12:49:22.339929 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.317358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.2.81.126:34751)
0924 12:49:22.318327 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:22.318340 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:22.319002 (+ 662us) server_negotiation.cc:407] Connection header received
0924 12:49:22.321488 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:22.321494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:22.321795 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:22.321999 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:22.323446 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.324729 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.326403 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.327127 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.327336 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:22.331455 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:22.331466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:22.331481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:22.331525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:22.334832 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:22.335631 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:22.335638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:22.335641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:22.335795 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:22.336210 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:22.336213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:22.336215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:22.336773 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:22.337007 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:22.337583 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 12:49:22.338045 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":757,"thread_start_us":549,"threads_started":1}
I20260924 12:49:22.344132 15896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:49:22.345719 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.344917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42494)
0924 12:49:22.345405 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.345538 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260924 12:49:22.346036 15846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
I20260924 12:49:22.348135 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.347385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42510)
0924 12:49:22.347737 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.347875 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260924 12:49:22.348593 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:33081: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab.
Entry for principal kudu/127.2.81.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab.
I20260924 12:49:22.375603 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:33081
--webserver_interface=127.2.81.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab
--principal=kudu/127.2.81.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:34751,127.2.81.125:33081,127.2.81.124:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:22.398041 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } attempt: 1
I20260924 12:49:22.399220 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.398746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42522)
0924 12:49:22.398949 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.399047 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 12:49:22.400293 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.399884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42528)
0924 12:49:22.400033 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.400136 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 12:49:22.400484 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:33081: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
I20260924 12:49:22.468757 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } attempt: 2
I20260924 12:49:22.470167 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.469636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42540)
0924 12:49:22.469906 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.470006 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 12:49:22.471366 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.470943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42544)
0924 12:49:22.471097 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.471225 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 12:49:22.471547 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:33081: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
I20260924 12:49:22.562867 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } attempt: 3
I20260924 12:49:22.564075 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.563571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42546)
0924 12:49:22.563836 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.563928 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 12:49:22.565228 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.564771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42558)
0924 12:49:22.564942 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.565065 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 12:49:22.565392 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:33081: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
I20260924 12:49:22.708570 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } attempt: 4
I20260924 12:49:22.709856 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.709346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42562)
0924 12:49:22.709638 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.709725 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 12:49:22.711089 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.710627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42576)
0924 12:49:22.710803 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.710923 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 12:49:22.711295 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:33081: Network error: Client connection negotiation failed: client connection to 127.2.81.125:33081: connect: Connection refused (error 111)
W20260924 12:49:22.731297 15903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:22.731716 15903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:22.731889 15903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:22.739624 15903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:22.739769 15903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:22.739871 15903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:22.739956 15903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:22.740017 15903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:22.751287 15903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:34751,127.2.81.125:33081,127.2.81.124:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab
--principal=kudu/127.2.81.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:33081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.81.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:22.752731 15903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:22.754181 15903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:22.766991 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:22.766952 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:22.766916 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:22.767793 15903 server_base.cc:1034] running on GCE node
I20260924 12:49:22.768618 15903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:22.770222 15903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:22.771484 15903 hybrid_clock.cc:648] HybridClock initialized: now 1790254162771430 us; error 41 us; skew 500 ppm
I20260924 12:49:22.771894 15903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:22 dist-test-slave-9tm1 krb5kdc[15816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:22.775832 15903 init.cc:377] Logged in from keytab as kudu/127.2.81.125@KRBTEST.COM (short username kudu)
I20260924 12:49:22.777760 15903 webserver.cc:460] Webserver started at http://127.2.81.125:37233/ using document root <none> and password file <none>
I20260924 12:49:22.778523 15903 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:22.778638 15903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:22.778962 15903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:22.782414 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data/instance:
uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577"
format_stamp: "Formatted at 2026-09-24 12:49:22 on dist-test-slave-9tm1"
server_key: "c770f5712303ac82b357bdb2e762b909"
server_key_iv: "88d46afd51496293b216521f7834dfe3"
server_key_version: "encryptionkey@0"
I20260924 12:49:22.783438 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal/instance:
uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577"
format_stamp: "Formatted at 2026-09-24 12:49:22 on dist-test-slave-9tm1"
server_key: "c770f5712303ac82b357bdb2e762b909"
server_key_iv: "88d46afd51496293b216521f7834dfe3"
server_key_version: "encryptionkey@0"
I20260924 12:49:22.790081 15903 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 12:49:22.794337 15918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:22.795912 15903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 12:49:22.796092 15903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal
uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577"
format_stamp: "Formatted at 2026-09-24 12:49:22 on dist-test-slave-9tm1"
server_key: "c770f5712303ac82b357bdb2e762b909"
server_key_iv: "88d46afd51496293b216521f7834dfe3"
server_key_version: "encryptionkey@0"
I20260924 12:49:22.796325 15903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:22.811951 15903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:22.816620 15903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:22.816910 15903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:22.835711 15903 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.125:33081
I20260924 12:49:22.835722 15969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.125:33081 every 8 connection(s)
I20260924 12:49:22.837303 15903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
I20260924 12:49:22.838724 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15903
I20260924 12:49:22.838927 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-1/wal/instance
I20260924 12:49:22.839445 2373 external_mini_cluster.cc:1428] Setting key ed5adf5b092986a8997d9798cd489323
I20260924 12:49:22.842635 15970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:22.851438 15970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.81.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.124@KRBTEST.COM" created.
Sep 24 12:49:22 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:22.874075 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.854524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.2.81.126:34751)
0924 12:49:22.854974 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:22.854979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:22.856362 (+ 1383us) server_negotiation.cc:407] Connection header received
0924 12:49:22.858991 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:22.858998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:22.859087 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:22.859188 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:22.860918 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.861968 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.863815 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.864287 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.864409 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:22.867956 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:22.867961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:22.867964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:22.868012 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:22.870600 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:22.871453 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:22.871458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:22.871460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:22.871591 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:22.872061 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:22.872065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:22.872067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:22.872371 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:22.872525 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:22.873420 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 12:49:22.873685 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
I20260924 12:49:22.875150 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.855313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:34751 (local address 127.0.0.1:36516)
0924 12:49:22.856170 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.856211 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:22.858630 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:22.859481 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:22.859491 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:22.860054 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:22.860694 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:22.860715 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.862209 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.862214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:22.863620 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:22.863637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.864547 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.864555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:22.864784 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:22.865477 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:22.865500 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:22.867691 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:22.870797 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:22.870808 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:22.870825 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:22.871308 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:22.871727 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:22.871732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:22.871735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:22.871936 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:22.872539 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:22.872549 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:22.872938 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 12:49:22.873293 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 12:49:22.873848 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":389,"threads_started":1}
I20260924 12:49:22.876932 15970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:49:22 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab.
Entry for principal kudu/127.2.81.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab.
I20260924 12:49:22.892695 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.877570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42588)
0924 12:49:22.877773 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.877792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:22.877898 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:22.879007 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:22.879010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:22.879395 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:22.879540 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:22.879548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.881580 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.881584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:22.882442 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:22.882455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.883442 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.883446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:22.883578 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:22.884184 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:22.884207 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:22.886818 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:22.890147 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:22.890152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:22.890156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:22.890547 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:22.890931 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:22.890936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:22.890940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:22.891100 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:22.891705 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:22.891710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:22.891862 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 12:49:22.891980 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 12:49:22.892283 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 12:49:22.892745 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.877818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.2.81.125:33081)
0924 12:49:22.878460 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:22.878469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:22.878502 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:49:22.878566 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:22.878574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:22.878833 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:22.879001 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:22.879703 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.881410 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.882599 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:22.883301 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:22.883400 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:22.887041 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:22.887055 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:22.887069 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:22.887108 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:22.890003 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:22.890688 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:22.890698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:22.890701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:22.890819 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:22.891201 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:22.891205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:22.891209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:22.891573 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:22.891769 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:22.892178 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 12:49:22.892397 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":304,"threads_started":1}
I20260924 12:49:22.894019 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:32857
--webserver_interface=127.2.81.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab
--principal=kudu/127.2.81.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:34751,127.2.81.125:33081,127.2.81.124:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:22.895442 15970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:49:22.896840 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.896318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49660)
0924 12:49:22.896546 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.896671 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260924 12:49:22.897187 15921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:22.898898 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.898515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49670)
0924 12:49:22.898681 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.898777 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 12:49:22.899123 15970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:22.948475 15970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 1
I20260924 12:49:22.949745 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.949267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49682)
0924 12:49:22.949473 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.949575 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 12:49:22.951036 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:22.950620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49686)
0924 12:49:22.950800 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:22.950907 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 12:49:22.951422 15970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.010579 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } attempt: 5
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:49:23.019716 15970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 2
I20260924 12:49:23.020744 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.020295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49690)
0924 12:49:23.020488 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.020577 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 12:49:23.021796 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.021416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49700)
0924 12:49:23.021596 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.021665 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 12:49:23.022082 15970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.026827 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.011310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42604)
0924 12:49:23.011525 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.011546 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:23.011659 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:23.012062 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:23.012066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:23.012495 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:23.012687 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.012697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.014500 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.014505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.015402 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.015420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.016473 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.016479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.016647 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.017331 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:23.017352 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:23.019782 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:23.023984 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.023989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.023992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.024434 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.024898 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.024902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.024905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.025129 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.025844 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:23.025848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:23.026001 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 12:49:23.026184 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 12:49:23.026555 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 12:49:23.026943 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.011456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.2.81.125:33081)
0924 12:49:23.011640 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.011657 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.011682 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:49:23.011751 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.011756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.011859 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.011971 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.012911 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.014279 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.015599 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.016263 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.016403 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.020097 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.020105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.020110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.020157 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.023781 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.024575 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.024581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.024584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.024750 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.025257 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.025263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.025266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.025672 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.025854 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.026319 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.026582 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 12:49:23.028263 15896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:49:23.029497 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.029087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49706)
0924 12:49:23.029258 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.029350 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 12:49:23.030700 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.030312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49710)
0924 12:49:23.030491 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.030570 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 12:49:23.030926 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.082226 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 1
I20260924 12:49:23.083400 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.082909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49720)
0924 12:49:23.083124 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.083227 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 12:49:23.084560 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.084189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49722)
0924 12:49:23.084361 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.084441 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 12:49:23.084810 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.113336 15970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 3
I20260924 12:49:23.114562 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.114056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49728)
0924 12:49:23.114300 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.114399 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 12:49:23.115648 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.115203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49732)
0924 12:49:23.115437 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.115518 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260924 12:49:23.115872 15970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.153098 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 2
I20260924 12:49:23.154350 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.153886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49746)
0924 12:49:23.154089 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.154189 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 12:49:23.155339 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.154999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49760)
0924 12:49:23.155167 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.155235 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 12:49:23.155550 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
W20260924 12:49:23.253714 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:23.254056 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:23.254204 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 12:49:23.259104 15970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 4
I20260924 12:49:23.260355 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.259847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49776)
0924 12:49:23.260084 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.260195 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 12:49:23.261494 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.261065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49782)
0924 12:49:23.261279 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.261365 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 12:49:23.261715 15970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.261730 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 3
I20260924 12:49:23.262836 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.262327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49784)
0924 12:49:23.262608 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.262691 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 12:49:23.263144 15978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:23.263257 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:23.263338 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:23.263420 15978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:23.263481 15978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:23.263834 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.263461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49800)
0924 12:49:23.263647 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.263725 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 12:49:23.264045 15896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:32857: Network error: Client connection negotiation failed: client connection to 127.2.81.124:32857: connect: Connection refused (error 111)
I20260924 12:49:23.274145 15978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:34751,127.2.81.125:33081,127.2.81.124:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab
--principal=kudu/127.2.81.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.81.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:23.275717 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:23.277801 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:23.290676 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:23.290742 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:23.290704 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:23.291224 15978 server_base.cc:1034] running on GCE node
I20260924 12:49:23.292131 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:23.293766 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:23.295028 15978 hybrid_clock.cc:648] HybridClock initialized: now 1790254163294969 us; error 52 us; skew 500 ppm
I20260924 12:49:23.295529 15978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:23.299885 15978 init.cc:377] Logged in from keytab as kudu/127.2.81.124@KRBTEST.COM (short username kudu)
I20260924 12:49:23.301869 15978 webserver.cc:460] Webserver started at http://127.2.81.124:45779/ using document root <none> and password file <none>
I20260924 12:49:23.302667 15978 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:23.302856 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:23.303239 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:23.307228 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data/instance:
uuid: "9a99218403344e41aa8240ca4ab5775a"
format_stamp: "Formatted at 2026-09-24 12:49:23 on dist-test-slave-9tm1"
server_key: "4e7f14a1734e4920b7ba0f10a27e7a7f"
server_key_iv: "d8e81d496c7850b8d372a6b635305d85"
server_key_version: "encryptionkey@0"
I20260924 12:49:23.308233 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal/instance:
uuid: "9a99218403344e41aa8240ca4ab5775a"
format_stamp: "Formatted at 2026-09-24 12:49:23 on dist-test-slave-9tm1"
server_key: "4e7f14a1734e4920b7ba0f10a27e7a7f"
server_key_iv: "d8e81d496c7850b8d372a6b635305d85"
server_key_version: "encryptionkey@0"
I20260924 12:49:23.314656 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 12:49:23.318842 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:23.320508 15978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:49:23.320699 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal
uuid: "9a99218403344e41aa8240ca4ab5775a"
format_stamp: "Formatted at 2026-09-24 12:49:23 on dist-test-slave-9tm1"
server_key: "4e7f14a1734e4920b7ba0f10a27e7a7f"
server_key_iv: "d8e81d496c7850b8d372a6b635305d85"
server_key_version: "encryptionkey@0"
I20260924 12:49:23.320928 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:23.335398 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:23.340204 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:23.340490 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:23.361851 15978 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.124:32857
I20260924 12:49:23.361866 16044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.124:32857 every 8 connection(s)
I20260924 12:49:23.363304 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
I20260924 12:49:23.367535 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 15978
I20260924 12:49:23.367717 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790253949676453-2373-0/minicluster-data/master-2/wal/instance
I20260924 12:49:23.368266 2373 external_mini_cluster.cc:1428] Setting key 64553e8b5964630a9d90253a88545055
I20260924 12:49:23.368549 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:23.380131 16045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:23.397578 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.370271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.2.81.124:32857)
0924 12:49:23.371126 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.371142 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.371195 (+ 53us) server_negotiation.cc:407] Connection header received
0924 12:49:23.373726 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.373757 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.375276 (+ 1519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.375979 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.378002 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.379582 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.381980 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.382925 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.383165 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.388086 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.388096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.388109 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.388156 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.391825 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.393494 (+ 1669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.393620 (+ 126us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.393635 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.393854 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.394379 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.394388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.394390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.395034 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.395270 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.395864 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.396385 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":406,"threads_started":1}
I20260924 12:49:23.401826 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.383658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.2.81.126:34751)
0924 12:49:23.384268 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.384275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.385058 (+ 783us) server_negotiation.cc:407] Connection header received
0924 12:49:23.385419 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.385423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.385509 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.385607 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.387061 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.387875 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.389944 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.390640 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.390841 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.395180 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.395188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.395192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.395237 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.398387 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.399352 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.399358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.399362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.399525 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.400061 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.400067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.400070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.400487 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.400631 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.401306 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.401511 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":321,"threads_started":1}
I20260924 12:49:23.401954 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.383642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:34751 (local address 127.0.0.1:36526)
0924 12:49:23.384881 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.384930 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:23.385228 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:23.385722 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:23.385730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:23.386566 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:23.386870 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.386881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.388155 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.388160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.389749 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.389768 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.390911 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.390917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.391060 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.391881 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:23.391908 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:23.394924 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:23.398545 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.398558 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.398566 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.399076 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.399651 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.399657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.399662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.399868 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.400621 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:23.400629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:23.400841 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 12:49:23.401078 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 12:49:23.401470 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":781,"threads_started":1}
I20260924 12:49:23.403836 16045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:49:23.419294 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.400904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.2.81.126:34751)
0924 12:49:23.402126 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.402132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.402146 (+ 14us) server_negotiation.cc:407] Connection header received
0924 12:49:23.402213 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.402217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.402307 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.402426 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.403892 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.404930 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.407116 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.407774 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.407910 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.413312 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.413318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.413321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.413360 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.416474 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.417324 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.417328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.417331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.417466 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.417925 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.417931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.417934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.418281 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.418527 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.418828 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.419016 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":1329,"threads_started":1}
I20260924 12:49:23.420369 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.404643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:33081 (local address 127.0.0.1:42618)
0924 12:49:23.404850 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.404872 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:23.405016 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:23.405560 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:23.405565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:23.406554 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:23.406830 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.406844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.408287 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.408293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.409684 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.409703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.410524 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.410531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.410796 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.411485 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:23.411512 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:23.414214 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:23.417555 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.417561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.417565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.418023 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.418504 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.418508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.418512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.418681 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.419351 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:23.419356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:23.419529 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 12:49:23.419676 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 12:49:23.419977 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 12:49:23.420395 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.404886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.2.81.125:33081)
0924 12:49:23.405117 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.405121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.405136 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:49:23.405187 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.405191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.405404 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.405487 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.407087 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.408128 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.409888 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.410338 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.410451 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.414447 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.414453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.414457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.414493 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.417393 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.418205 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.418209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.418212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.418368 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.418814 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.418819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.418822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.419157 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.419319 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.419816 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.420024 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 12:49:23.421653 16045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:49:23.435973 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.422489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49818)
0924 12:49:23.422687 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.422703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:23.422780 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:23.423089 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:23.423092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:23.423537 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:23.423746 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.423758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.424768 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.424772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.425847 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.425859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.426740 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.426746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.426971 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.427674 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:23.427696 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:23.430155 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:23.433306 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.433312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.433316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.433812 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.434183 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.434188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.434191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.434363 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.435000 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:23.435003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:23.435161 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 12:49:23.435309 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 12:49:23.435589 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 12:49:23.436048 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.422575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.2.81.124:32857)
0924 12:49:23.422761 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.422767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.422785 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:49:23.422859 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.422863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.422933 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.423037 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.423870 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.424617 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.425971 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.426576 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.426677 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.430379 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.430385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.430390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.430421 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.433163 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.433952 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.433956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.433958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.434070 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.434480 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.434484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.434487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.434877 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.435024 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.435417 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.435807 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":84}
I20260924 12:49:23.437350 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.421404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.2.81.125:33081)
0924 12:49:23.421580 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.421585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.421601 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:49:23.421677 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.421680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.421753 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.421825 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.423463 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.424368 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.425888 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.426510 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.426648 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.431217 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.431223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.431226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.431258 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.434236 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.435126 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.435131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.435134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.435323 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.435753 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.435758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.435762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.436083 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.436363 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.436773 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.437038 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:49:23.441272 15896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 4
I20260924 12:49:23.441514 16045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: Bootstrap starting.
I20260924 12:49:23.442446 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:49:23.445329 16045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:23.446596 16045 log.cc:824] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
I20260924 12:49:23.456676 16045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: No bootstrap required, opened a new log
I20260924 12:49:23.460072 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.441864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49830)
0924 12:49:23.442060 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.442081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:23.442192 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:23.442586 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:23.442591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:23.443033 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:23.443236 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.443261 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.444632 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.444637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.446159 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.446179 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.447244 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.447258 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.447450 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.449183 (+ 1733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:23.449225 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:23.452321 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:23.457064 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.457070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.457073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.457618 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.458121 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.458127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.458130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.458326 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.459039 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:23.459044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:23.459222 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 12:49:23.459347 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 12:49:23.459669 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 12:49:23.461601 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.442100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.2.81.124:32857)
0924 12:49:23.442283 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.442289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.442306 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:49:23.442356 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.442360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.442430 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.442527 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.443416 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.444458 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.446338 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.447047 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.447239 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.452627 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.452635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.452641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.452688 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.456830 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.457805 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.457810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.457813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.457983 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.458478 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.458483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.458486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.458875 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.460799 (+ 1924us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.460953 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.461232 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 12:49:23.467824 15896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: Bootstrap starting.
I20260924 12:49:23.468086 16045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.468676 16045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:23.468866 16045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a99218403344e41aa8240ca4ab5775a, State: Initialized, Role: FOLLOWER
I20260924 12:49:23.469760 16045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.472579 15896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:23.474570 15896 log.cc:824] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:23.477125 16060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.480542 16060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.479455 16045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:23.483283 15896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: No bootstrap required, opened a new log
I20260924 12:49:23.491084 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.448447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.2.81.126:34751)
0924 12:49:23.448671 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.448676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.461408 (+ 12732us) server_negotiation.cc:407] Connection header received
0924 12:49:23.462492 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.462499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.462587 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.462662 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.467902 (+ 5240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.469328 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.474216 (+ 4888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.474996 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.475117 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.482597 (+ 7480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.482606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.482611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.482657 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.486724 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.487910 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.487916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.487920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.488106 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.489003 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.489008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.489011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.489421 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.489726 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.490391 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.490678 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:49:23.499467 15896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.500082 15896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:23.500267 15896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1ff4d8500e4b7fa6af1166ce6c787e, State: Initialized, Role: FOLLOWER
I20260924 12:49:23.500856 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.476026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.2.81.125:33081)
0924 12:49:23.476203 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.476209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.483072 (+ 6863us) server_negotiation.cc:407] Connection header received
0924 12:49:23.483545 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.483551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.483649 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.483722 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.486444 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.487561 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.489716 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.490333 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.490430 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.493953 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.493960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.493965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.494018 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.496650 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.497594 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.497600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.497604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.497785 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.498269 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.498274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.498277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.498636 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.498944 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.500128 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.500502 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:49:23.501225 15896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.507983 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.448856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.2.81.124:32857)
0924 12:49:23.473003 (+ 24147us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.473010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.475935 (+ 2925us) server_negotiation.cc:407] Connection header received
0924 12:49:23.476245 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.476251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.476344 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.476418 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.479915 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.480917 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.497186 (+ 16269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.497896 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.497998 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.501257 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.501264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.501269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.501314 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.505012 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.505858 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.505863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.505865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.506002 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.506403 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.506408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.506410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.506733 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.507027 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.507335 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.507572 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23988,"thread_start_us":40888,"threads_started":1}
I20260924 12:49:23.508443 16067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.511050 16067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.510907 15896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 12:49:23.519240 16074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:49:23.519508 16074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:49:23.514847 15978 master_runner.cc:186] Error getting master registration for 127.2.81.126:34751: OK, instance_id { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" instance_seqno: 1790254162290314 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 12:49:23.526628 15978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 12:49:23.527968 15850 tablet.cc:2369] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:49:23.531478 16086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:49:23.531661 16086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:49:23.531172 15829 master_runner.cc:186] Error getting master registration for 127.2.81.125:33081: OK, instance_id { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" instance_seqno: 1790254162829996 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 12:49:23.542616 15829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 12:49:23.560993 15970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } attempt: 5
Sep 24 12:49:23 dist-test-slave-9tm1 krb5kdc[15816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
I20260924 12:49:23.578832 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.561655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:32857 (local address 127.0.0.1:49846)
0924 12:49:23.562004 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:23.562025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:23.562145 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:23.562642 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:23.562648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:23.563166 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:23.563392 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.563405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.564859 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.564865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.566291 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:23.566307 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.567393 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.567399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:23.567644 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.568367 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:23.568395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:23.571390 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:23.575760 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.575766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.575771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.576373 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.576850 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:23.576856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:23.576860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:23.577076 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:23.577717 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:23.577723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:23.577890 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 12:49:23.578104 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 12:49:23.578434 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260924 12:49:23.578888 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:23.561750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.2.81.124:32857)
0924 12:49:23.562300 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:23.562307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:23.562327 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:49:23.562386 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:23.562391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:23.562472 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:23.562586 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:23.563565 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.564689 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.566455 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:23.567218 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:23.567353 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:23.571612 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:23.571619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:23.571624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:23.571666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:23.575553 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.576522 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.576528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.576531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.576713 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:23.577192 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:23.577196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:23.577203 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:23.577583 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:23.577730 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:23.578197 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 12:49:23.578470 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260924 12:49:23.585161 15970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: Bootstrap starting.
I20260924 12:49:23.589358 15970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:23.590759 15970 log.cc:824] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: Log is configured to *not* fsync() on all Append() calls
W20260924 12:49:23.592934 15999 tablet.cc:2369] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:23.594523 15970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: No bootstrap required, opened a new log
I20260924 12:49:23.606621 15970 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.607183 15970 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:23.607321 15970 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f18cb5b4ee4d8ebf38a21a4ee7b577, State: Initialized, Role: FOLLOWER
I20260924 12:49:23.608237 15970 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.612057 16089 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.612565 16089 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.613788 15970 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:23.631604 15903 master_runner.cc:263] No leader master found from master a8f18cb5b4ee4d8ebf38a21a4ee7b577
I20260924 12:49:23.631784 15903 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 12:49:23.639978 16100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:49:23.640275 16100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:49:23.675868 16067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:49:23.676249 16067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.678215 16067 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f18cb5b4ee4d8ebf38a21a4ee7b577 (127.2.81.125:33081), 9a99218403344e41aa8240ca4ab5775a (127.2.81.124:32857)
I20260924 12:49:23.679484 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a99218403344e41aa8240ca4ab5775a" is_pre_election: true
I20260924 12:49:23.680183 16020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1ff4d8500e4b7fa6af1166ce6c787e in term 0.
I20260924 12:49:23.681161 15847 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1ff4d8500e4b7fa6af1166ce6c787e, 9a99218403344e41aa8240ca4ab5775a; no voters:
I20260924 12:49:23.682768 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" is_pre_election: true
I20260924 12:49:23.683426 15945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1ff4d8500e4b7fa6af1166ce6c787e in term 0.
I20260924 12:49:23.688347 16067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:49:23.688563 16067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:49:23.688663 16067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:23.693034 16067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.694592 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577"
I20260924 12:49:23.694845 15945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:23.698561 15945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1ff4d8500e4b7fa6af1166ce6c787e in term 1.
I20260924 12:49:23.699204 15846 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1ff4d8500e4b7fa6af1166ce6c787e, a8f18cb5b4ee4d8ebf38a21a4ee7b577; no voters:
I20260924 12:49:23.702323 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a99218403344e41aa8240ca4ab5775a"
I20260924 12:49:23.702597 16020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:23.706454 16020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1ff4d8500e4b7fa6af1166ce6c787e in term 1.
I20260924 12:49:23.712018 16067 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [CANDIDATE]: Term 1 election: Requested vote from peers a8f18cb5b4ee4d8ebf38a21a4ee7b577 (127.2.81.125:33081), 9a99218403344e41aa8240ca4ab5775a (127.2.81.124:32857)
I20260924 12:49:23.712575 16067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:23.713166 16067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [term 1 LEADER]: Becoming Leader. State: Replica: 5c1ff4d8500e4b7fa6af1166ce6c787e, State: Running, Role: LEADER
I20260924 12:49:23.714823 16067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } }
I20260924 12:49:23.726015 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1ff4d8500e4b7fa6af1166ce6c787e. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.726694 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:23.731647 16104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:23.734242 16104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:23.756991 16020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [term 1 FOLLOWER]: Refusing update from remote peer 5c1ff4d8500e4b7fa6af1166ce6c787e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:23.758291 16102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:23.766741 15945 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [term 1 FOLLOWER]: Refusing update from remote peer 5c1ff4d8500e4b7fa6af1166ce6c787e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:23.771595 16102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:23.791682 16060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1ff4d8500e4b7fa6af1166ce6c787e. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.792049 16060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.804840 16105 mvcc.cc:205] Tried to move back new op lower bound from 7332881054728372224 to 7332881054590939136. Current Snapshot: MvccSnapshot[applied={T|T < 7332881054728372224}]
I20260924 12:49:23.805233 16106 mvcc.cc:205] Tried to move back new op lower bound from 7332881054728372224 to 7332881054590939136. Current Snapshot: MvccSnapshot[applied={T|T < 7332881054728372224}]
I20260924 12:49:23.818233 16067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.818638 16067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: This master's current role is: LEADER
W20260924 12:49:23.820084 15924 tablet.cc:2369] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:23.825430 16060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.825737 16060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.833323 16089 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1ff4d8500e4b7fa6af1166ce6c787e. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.833686 16089 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.840796 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.841730 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:23.845206 16107 mvcc.cc:205] Tried to move back new op lower bound from 7332881054728372224 to 7332881054590939136. Current Snapshot: MvccSnapshot[applied={T|T < 7332881054728372224}]
I20260924 12:49:23.849030 16104 catalog_manager.cc:1347] Generated new cluster ID: 4e731e36797f49b99436a52a87189e0c
I20260924 12:49:23.849185 16104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:23.865065 16089 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1ff4d8500e4b7fa6af1166ce6c787e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 34751 } } peers { permanent_uuid: "a8f18cb5b4ee4d8ebf38a21a4ee7b577" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 33081 } } peers { permanent_uuid: "9a99218403344e41aa8240ca4ab5775a" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 32857 } } }
I20260924 12:49:23.865367 16089 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:49:23.892764 16104 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:49:23.894842 16104 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:23.923460 16104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1ff4d8500e4b7fa6af1166ce6c787e: Generated new TSK 0
I20260924 12:49:23.924659 16104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:49:24.521783 16074 catalog_manager.cc:1259] Loaded cluster ID: 4e731e36797f49b99436a52a87189e0c
I20260924 12:49:24.521893 16074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: loading cluster ID for follower catalog manager: success
I20260924 12:49:24.524456 16074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: acquiring CA information for follower catalog manager: success
I20260924 12:49:24.526428 16074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a99218403344e41aa8240ca4ab5775a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 12:49:24.643020 16100 catalog_manager.cc:1259] Loaded cluster ID: 4e731e36797f49b99436a52a87189e0c
I20260924 12:49:24.643157 16100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: loading cluster ID for follower catalog manager: success
I20260924 12:49:24.646386 16100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: acquiring CA information for follower catalog manager: success
I20260924 12:49:24.648416 16100 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8f18cb5b4ee4d8ebf38a21a4ee7b577: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 12:49:25.074810 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15829
I20260924 12:49:25.076130 15891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:25.214066 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15829
I20260924 12:49:25.235596 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15903
I20260924 12:49:25.236874 15965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:25.397554 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15903
I20260924 12:49:25.418421 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 15978
I20260924 12:49:25.419838 16040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:25.532393 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 15978
2026-09-24T12:49:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5047 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:49:25 dist-test-slave-9tm1 krb5kdc[16123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:49:25 dist-test-slave-9tm1 krb5kdc[16123](info): set up 2 sockets
Sep 24 12:49:25 dist-test-slave-9tm1 krb5kdc[16123](info): commencing operation
krb5kdc: starting...
W20260924 12:49:26.601620 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:49:26 dist-test-slave-9tm1 krb5kdc[16123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:49:26.845739 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33829
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:27.191552 16136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:27.192414 16136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:27.192704 16136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:27.201856 16136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:27.202028 16136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:27.202102 16136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:27.202183 16136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:27.202253 16136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:27.213253 16136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:33829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:27.214663 16136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:27.216018 16136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:27.229843 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:27.229871 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:27.229933 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:27.231189 16136 server_base.cc:1034] running on GCE node
I20260924 12:49:27.232184 16136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:27.233904 16136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:27.235189 16136 hybrid_clock.cc:648] HybridClock initialized: now 1790254167235137 us; error 38 us; skew 500 ppm
I20260924 12:49:27.235762 16136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:27 dist-test-slave-9tm1 krb5kdc[16123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:27.243328 16136 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:27.245048 16136 webserver.cc:460] Webserver started at http://127.2.81.126:40543/ using document root <none> and password file <none>
I20260924 12:49:27.245967 16136 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:27.246114 16136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:27.246436 16136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:27.250811 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "5bb00b398e4c4006b7be28e57eb40fe8"
format_stamp: "Formatted at 2026-09-24 12:49:27 on dist-test-slave-9tm1"
server_key: "307217c3844207285061c9ff03b95407"
server_key_iv: "0b949c488584b4f0e878d62563faee9b"
server_key_version: "encryptionkey@0"
I20260924 12:49:27.252230 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "5bb00b398e4c4006b7be28e57eb40fe8"
format_stamp: "Formatted at 2026-09-24 12:49:27 on dist-test-slave-9tm1"
server_key: "307217c3844207285061c9ff03b95407"
server_key_iv: "0b949c488584b4f0e878d62563faee9b"
server_key_version: "encryptionkey@0"
I20260924 12:49:27.259133 16136 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 12:49:27.265472 16151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:27.267491 16136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:49:27.267741 16136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "5bb00b398e4c4006b7be28e57eb40fe8"
format_stamp: "Formatted at 2026-09-24 12:49:27 on dist-test-slave-9tm1"
server_key: "307217c3844207285061c9ff03b95407"
server_key_iv: "0b949c488584b4f0e878d62563faee9b"
server_key_version: "encryptionkey@0"
I20260924 12:49:27.268018 16136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:27.303877 16136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:27.309722 16136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:27.309991 16136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:27.329339 16136 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:33829
I20260924 12:49:27.329381 16202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:33829 every 8 connection(s)
I20260924 12:49:27.330740 16136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:27.334842 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:27.339676 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16136
I20260924 12:49:27.339890 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:49:27.340576 2373 external_mini_cluster.cc:1428] Setting key 1a583de9ae682d027a4be3d529937e2d
I20260924 12:49:27.346522 16203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8: Bootstrap starting.
I20260924 12:49:27.350508 16203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:27.351953 16203 log.cc:824] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:27.356278 16203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8: No bootstrap required, opened a new log
Sep 24 12:49:27 dist-test-slave-9tm1 krb5kdc[16123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:27.365595 16203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33829 } }
I20260924 12:49:27.366045 16203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:27.366164 16203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb00b398e4c4006b7be28e57eb40fe8, State: Initialized, Role: FOLLOWER
I20260924 12:49:27.366945 16203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33829 } }
I20260924 12:49:27.367230 16203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:27.367375 16203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:27.367548 16203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:27.368494 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:27.342571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.2.81.126:33829)
0924 12:49:27.343479 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:27.343492 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:27.343541 (+ 49us) server_negotiation.cc:407] Connection header received
0924 12:49:27.345422 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:27.345448 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:27.345898 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:27.346270 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:27.348353 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:27.349889 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:27.351888 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:27.353035 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:27.353318 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:27.359054 (+ 5736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:27.359064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:27.359084 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:27.359149 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:27.363214 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:27.364387 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:27.364395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:27.364403 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:27.364652 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:27.365182 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:27.365187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:27.365189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:27.365827 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:27.366117 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:27.366637 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 12:49:27.367268 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":399,"threads_started":1}
I20260924 12:49:27.370155 16203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33829 } }
I20260924 12:49:27.370703 16203 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bb00b398e4c4006b7be28e57eb40fe8; no voters:
I20260924 12:49:27.371392 16203 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:49:27.371546 16208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:27.372758 16208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb00b398e4c4006b7be28e57eb40fe8, State: Running, Role: LEADER
I20260924 12:49:27.373081 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:49:27.373417 16208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33829 } }
I20260924 12:49:27.373730 16203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:27.378981 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb00b398e4c4006b7be28e57eb40fe8. Latest consensus state: current_term: 1 leader_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33829 } } }
I20260924 12:49:27.379016 16209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb00b398e4c4006b7be28e57eb40fe8" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 33829 } } }
I20260924 12:49:27.379395 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:27.379431 16209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8 [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:27.380463 16215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:27.388146 16215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:27.402977 16215 catalog_manager.cc:1347] Generated new cluster ID: 5d79d956dd544a56aa0815843edea82c
I20260924 12:49:27.403115 16215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:27.429559 16215 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:49:27.431005 16215 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:27.441839 16215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bb00b398e4c4006b7be28e57eb40fe8: Generated new TSK 0
I20260924 12:49:27.442849 16215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:49:27.486168 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16136
I20260924 12:49:27.487460 16198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:27.585767 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16136
2026-09-24T12:49:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2049 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:49:27 dist-test-slave-9tm1 krb5kdc[16231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:49:27 dist-test-slave-9tm1 krb5kdc[16231](info): set up 2 sockets
Sep 24 12:49:27 dist-test-slave-9tm1 krb5kdc[16231](info): commencing operation
krb5kdc: starting...
W20260924 12:49:28.641541 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.015s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:49:28 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:49:28.883044 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45513
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:29.217308 16244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:29.217613 16244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:29.217705 16244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:29.225103 16244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:29.225227 16244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:29.225273 16244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:29.225322 16244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:29.225370 16244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:29.235481 16244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:29.236783 16244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:29.238092 16244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:29.250293 16249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:29.250321 16250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:29.250356 16252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:29.251108 16244 server_base.cc:1034] running on GCE node
I20260924 12:49:29.251963 16244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:29.253541 16244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:29.254766 16244 hybrid_clock.cc:648] HybridClock initialized: now 1790254169254702 us; error 61 us; skew 500 ppm
I20260924 12:49:29.255131 16244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:29 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:29.258504 16244 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:29.260149 16244 webserver.cc:460] Webserver started at http://127.2.81.126:42509/ using document root <none> and password file <none>
I20260924 12:49:29.260878 16244 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:29.261023 16244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:29.261353 16244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:29.264428 16244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "ae6eb72f383549f3adbffd8169382c8b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "13b4cdd42556004908ba0070042465a9"
server_key_iv: "a854075f526dfaec9b2485090e69533c"
server_key_version: "encryptionkey@0"
I20260924 12:49:29.265273 16244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "ae6eb72f383549f3adbffd8169382c8b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "13b4cdd42556004908ba0070042465a9"
server_key_iv: "a854075f526dfaec9b2485090e69533c"
server_key_version: "encryptionkey@0"
I20260924 12:49:29.270725 16244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:49:29.274662 16259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:29.276368 16244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:49:29.276579 16244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "ae6eb72f383549f3adbffd8169382c8b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "13b4cdd42556004908ba0070042465a9"
server_key_iv: "a854075f526dfaec9b2485090e69533c"
server_key_version: "encryptionkey@0"
I20260924 12:49:29.276790 16244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:29.292829 16244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:29.298605 16244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:29.299084 16244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:29.318365 16244 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45513
I20260924 12:49:29.318387 16310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45513 every 8 connection(s)
I20260924 12:49:29.320225 16244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:29.326383 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16244
I20260924 12:49:29.326779 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:49:29.327778 2373 external_mini_cluster.cc:1428] Setting key 399ee7fe0f7c2a6322902a5a2e0e4f83
I20260924 12:49:29.329524 16311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:49:29 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:29.345741 16311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap starting.
I20260924 12:49:29.350080 16311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:29.351406 16311 log.cc:824] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:29.356366 16311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: No bootstrap required, opened a new log
I20260924 12:49:29.357211 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:29.330398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.2.81.126:45513)
0924 12:49:29.331230 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:29.331242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:29.331284 (+ 42us) server_negotiation.cc:407] Connection header received
0924 12:49:29.333055 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:29.333081 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:29.333586 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:29.333973 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:29.336528 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:29.338107 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:29.340272 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:29.341215 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:29.341406 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:29.346340 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:29.346353 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:29.346374 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:29.346428 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:29.350546 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:29.351542 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:29.351553 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:29.351561 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:29.351818 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:29.352726 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:29.352730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:29.352733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:29.353531 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:29.353864 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:29.354319 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 12:49:29.355427 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":383,"threads_started":1}
I20260924 12:49:29.365434 16311 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:29.365887 16311 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:29.365964 16311 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6eb72f383549f3adbffd8169382c8b, State: Initialized, Role: FOLLOWER
I20260924 12:49:29.366652 16311 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:29.366938 16311 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:29.367105 16311 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:29.367331 16311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:29.369764 16311 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:29.370307 16311 leader_election.cc:302] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6eb72f383549f3adbffd8169382c8b; no voters:
I20260924 12:49:29.371038 16311 leader_election.cc:288] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:49:29.371131 16316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:29.371908 16316 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 LEADER]: Becoming Leader. State: Replica: ae6eb72f383549f3adbffd8169382c8b, State: Running, Role: LEADER
I20260924 12:49:29.372570 16316 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:29.373654 16311 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:29.375572 16318 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6eb72f383549f3adbffd8169382c8b. Latest consensus state: current_term: 1 leader_uuid: "ae6eb72f383549f3adbffd8169382c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } } }
I20260924 12:49:29.375900 16318 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:29.376808 16322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:29.376559 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6eb72f383549f3adbffd8169382c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } } }
I20260924 12:49:29.377701 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:29.383126 16322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:29.395872 16322 catalog_manager.cc:1347] Generated new cluster ID: e3ab8046f39b43439840e3232cf04f00
I20260924 12:49:29.395995 16322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:29.413617 16322 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:49:29.415192 16322 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:29.427891 16322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Generated new TSK 0
I20260924 12:49:29.430208 16322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:49:29.514029 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45513
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:29.895120 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:29.895429 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:29.895565 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:29.903688 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:29.903868 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:29.904029 16337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:49:29.915669 16337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:29.917078 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:29.918442 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:29.932147 16342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:29.932250 16345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:29.932164 16343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:29.933002 16337 server_base.cc:1034] running on GCE node
I20260924 12:49:29.933712 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:29.934870 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:29.936139 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790254169936084 us; error 52 us; skew 500 ppm
I20260924 12:49:29.936506 16337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:29 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:29.940552 16337 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:49:29.942564 16337 webserver.cc:460] Webserver started at http://127.2.81.65:45061/ using document root <none> and password file <none>
I20260924 12:49:29.943377 16337 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:29.943514 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:29.943819 16337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:29.947018 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "59e62a47082a493090195cfec125a64b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "8868f5248ac4340d38539ead50fbd576"
server_key_iv: "b17b0157af8080e5aeba23a6eab884f0"
server_key_version: "encryptionkey@0"
I20260924 12:49:29.947885 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "59e62a47082a493090195cfec125a64b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "8868f5248ac4340d38539ead50fbd576"
server_key_iv: "b17b0157af8080e5aeba23a6eab884f0"
server_key_version: "encryptionkey@0"
I20260924 12:49:29.953583 16337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 12:49:29.958201 16352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:29.960675 16337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:49:29.960901 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "59e62a47082a493090195cfec125a64b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "8868f5248ac4340d38539ead50fbd576"
server_key_iv: "b17b0157af8080e5aeba23a6eab884f0"
server_key_version: "encryptionkey@0"
I20260924 12:49:29.961184 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:29.983422 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:29.987967 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:29.988274 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:29.989354 16337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:29.990867 16337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:29.990965 16337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:29.991096 16337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:29.991148 16337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:30.027873 16337 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:33839
I20260924 12:49:30.027948 16464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:33839 every 8 connection(s)
I20260924 12:49:30.029348 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:49:30.032612 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16337
I20260924 12:49:30.032822 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:49:30.033536 2373 external_mini_cluster.cc:1428] Setting key a242df0ea0ee1e271279b4877ad1ff5c
Sep 24 12:49:30 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:30.057581 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:30.034302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:44915 (local address 127.2.81.126:45513)
0924 12:49:30.038249 (+ 3947us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:30.038255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:30.038275 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:49:30.038337 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:30.038341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:30.038421 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:30.038539 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:30.041209 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.042259 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.044062 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.044488 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.044618 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:30.049275 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:30.049283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:30.049288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:30.049345 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:30.052943 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:30.054318 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:30.054324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:30.054327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:30.054504 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:30.055299 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:30.055306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:30.055310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:30.055718 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:30.055932 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:30.056853 (+ 921us) server_negotiation.cc:299] Negotiation successful
0924 12:49:30.057173 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3814,"thread_start_us":3667,"threads_started":1}
I20260924 12:49:30.060704 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:30.033648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:44915)
0924 12:49:30.034690 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:30.034754 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:30.037566 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:30.039073 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:30.039089 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:30.039895 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:30.040882 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:30.040913 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.042501 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.042506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:30.043852 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:30.043869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.044768 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.044776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:30.045118 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:30.046039 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:30.046076 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:30.048922 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:30.053210 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:30.053222 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:30.053244 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:30.054118 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:30.054844 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:30.054848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:30.054851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:30.055088 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:30.055958 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:30.055965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:30.056380 (+ 415us) client_negotiation.cc:769] Sending connection context
0924 12:49:30.056647 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 12:49:30.057460 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":377,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:49:30.062927 16465 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45513
I20260924 12:49:30.063455 16465 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:30.064572 16465 heartbeater.cc:511] Master 127.2.81.126:45513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:49:30.067240 16276 ts_manager.cc:194] Registered new tserver with Master: 59e62a47082a493090195cfec125a64b (127.2.81.65:33839)
I20260924 12:49:30.068813 16276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:44915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:49:30.105996 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45513
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:30.447885 16471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:30.448174 16471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:30.448331 16471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:30.455806 16471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:30.455955 16471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:30.456117 16471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:49:30.466490 16471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:30.467796 16471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:30.469166 16471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:30.482841 16479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:30.482915 16476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:30.482879 16477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:30.483839 16471 server_base.cc:1034] running on GCE node
I20260924 12:49:30.484490 16471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:30.485517 16471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:30.486742 16471 hybrid_clock.cc:648] HybridClock initialized: now 1790254170486697 us; error 36 us; skew 500 ppm
I20260924 12:49:30.487064 16471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:30 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:30.490123 16471 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:49:30.491696 16471 webserver.cc:460] Webserver started at http://127.2.81.66:42337/ using document root <none> and password file <none>
I20260924 12:49:30.492388 16471 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:30.492504 16471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:30.492812 16471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:30.495843 16471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "b33fae30b3364a09b4f6bd446833b7f3"
format_stamp: "Formatted at 2026-09-24 12:49:30 on dist-test-slave-9tm1"
server_key: "6e174ee62e2afa48bb063fe0eaa2978d"
server_key_iv: "c6a2758e9d09f5b2bc8ddcea6bab6a93"
server_key_version: "encryptionkey@0"
I20260924 12:49:30.496585 16471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "b33fae30b3364a09b4f6bd446833b7f3"
format_stamp: "Formatted at 2026-09-24 12:49:30 on dist-test-slave-9tm1"
server_key: "6e174ee62e2afa48bb063fe0eaa2978d"
server_key_iv: "c6a2758e9d09f5b2bc8ddcea6bab6a93"
server_key_version: "encryptionkey@0"
I20260924 12:49:30.501926 16471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:49:30.505419 16486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:30.506781 16471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:49:30.506942 16471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "b33fae30b3364a09b4f6bd446833b7f3"
format_stamp: "Formatted at 2026-09-24 12:49:30 on dist-test-slave-9tm1"
server_key: "6e174ee62e2afa48bb063fe0eaa2978d"
server_key_iv: "c6a2758e9d09f5b2bc8ddcea6bab6a93"
server_key_version: "encryptionkey@0"
I20260924 12:49:30.507128 16471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:30.535377 16471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:30.539557 16471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:30.539834 16471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:30.540786 16471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:30.542181 16471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:30.542266 16471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:30.542385 16471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:30.542438 16471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:30.574268 16471 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:35689
I20260924 12:49:30.574316 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:35689 every 8 connection(s)
I20260924 12:49:30.575584 16471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:49:30.579290 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16471
I20260924 12:49:30.579514 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:49:30.580080 2373 external_mini_cluster.cc:1428] Setting key 443d64cc0400d062912c15cac088bda7
Sep 24 12:49:30 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254170, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:30.600389 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:30.578088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:51391 (local address 127.2.81.126:45513)
0924 12:49:30.578550 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:30.578555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:30.579655 (+ 1100us) server_negotiation.cc:407] Connection header received
0924 12:49:30.581965 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:30.581970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:30.582051 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:30.582166 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:30.584341 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.585143 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.587529 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.588029 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.588148 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:30.592738 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:30.592746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:30.592751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:30.592798 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:30.596271 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:30.597457 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:30.597464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:30.597468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:30.597779 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:30.598431 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:30.598437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:30.598440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:30.598794 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:30.598936 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:30.599732 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 12:49:30.600000 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260924 12:49:30.601459 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:30.578493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:51391)
0924 12:49:30.579419 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:30.579476 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:30.581681 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:30.582421 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:30.582434 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:30.583266 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:30.584105 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:30.584134 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.585294 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.585300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:30.587341 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:30.587359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:30.588276 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:30.588283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:30.588596 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:30.589443 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:30.589474 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:30.592406 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:30.596502 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:30.596515 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:30.596536 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:30.597298 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:30.598028 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:30.598034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:30.598038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:30.598282 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:30.598949 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:30.598957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:30.599330 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 12:49:30.599591 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 12:49:30.600170 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":351,"threads_started":1}
I20260924 12:49:30.603076 16599 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45513
I20260924 12:49:30.603406 16599 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:30.604154 16599 heartbeater.cc:511] Master 127.2.81.126:45513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:49:30.605892 16276 ts_manager.cc:194] Registered new tserver with Master: b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689)
I20260924 12:49:30.606789 16276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:51391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:49:30.637573 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45513
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:30.964227 16605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:30.964529 16605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:30.964641 16605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:30.972251 16605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:30.972395 16605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:30.972546 16605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:49:30.985118 16605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:30.986481 16605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:30.987860 16605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:31.002221 16611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:31.002327 16613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:31.002235 16610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:31.003023 16605 server_base.cc:1034] running on GCE node
I20260924 12:49:31.003737 16605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:31.004873 16605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:31.006129 16605 hybrid_clock.cc:648] HybridClock initialized: now 1790254171006062 us; error 58 us; skew 500 ppm
I20260924 12:49:31.006516 16605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:31 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:31.010044 16605 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:49:31.011900 16605 webserver.cc:460] Webserver started at http://127.2.81.67:32901/ using document root <none> and password file <none>
I20260924 12:49:31.012717 16605 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:31.013015 16605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:31.013406 16605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:31.016533 16605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "196b120126024e6e9d21749756f1484e"
format_stamp: "Formatted at 2026-09-24 12:49:31 on dist-test-slave-9tm1"
server_key: "822afad6cf931734f50f79bbeda62e5d"
server_key_iv: "64e040d50e688ea885874c44dc0d0dfa"
server_key_version: "encryptionkey@0"
I20260924 12:49:31.017467 16605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "196b120126024e6e9d21749756f1484e"
format_stamp: "Formatted at 2026-09-24 12:49:31 on dist-test-slave-9tm1"
server_key: "822afad6cf931734f50f79bbeda62e5d"
server_key_iv: "64e040d50e688ea885874c44dc0d0dfa"
server_key_version: "encryptionkey@0"
I20260924 12:49:31.023119 16605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:49:31.027292 16620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:31.028914 16605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:49:31.029115 16605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "196b120126024e6e9d21749756f1484e"
format_stamp: "Formatted at 2026-09-24 12:49:31 on dist-test-slave-9tm1"
server_key: "822afad6cf931734f50f79bbeda62e5d"
server_key_iv: "64e040d50e688ea885874c44dc0d0dfa"
server_key_version: "encryptionkey@0"
I20260924 12:49:31.029336 16605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:31.054140 16605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:31.059517 16605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:31.059862 16605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:31.060958 16605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:31.063184 16605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:31.063292 16605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:31.063434 16605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:31.063493 16605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:31.073904 16465 heartbeater.cc:503] Master 127.2.81.126:45513 was elected leader, sending a full tablet report...
I20260924 12:49:31.099080 16605 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:35969
I20260924 12:49:31.099123 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:35969 every 8 connection(s)
I20260924 12:49:31.100382 16605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:49:31.110541 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16605
I20260924 12:49:31.110725 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:49:31.111191 2373 external_mini_cluster.cc:1428] Setting key a800d0fce5b93d1edf255391c78c0477
Sep 24 12:49:31 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254171, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:31.122710 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.103162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:53107 (local address 127.2.81.126:45513)
0924 12:49:31.103826 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.103832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.104951 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 12:49:31.107000 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.107005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.107088 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.107200 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:31.109143 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.110082 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.112041 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.112637 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.112770 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.116701 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:31.116708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:31.116713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:31.116754 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:31.119578 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.120368 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.120373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.120375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.120503 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.121022 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.121027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.121031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.121375 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:31.121516 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.122216 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.122438 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
I20260924 12:49:31.123699 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.103913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:53107)
0924 12:49:31.104736 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:31.104785 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:31.106611 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:31.107530 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:31.107540 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:31.108148 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:31.108800 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.108829 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.110316 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.110321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.111866 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.111886 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.112926 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.112936 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.113266 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.114019 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:31.114044 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:31.116463 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:31.119765 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:31.119773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:31.119788 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:31.120244 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:31.120643 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:31.120647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:31.120650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:31.120868 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:31.121508 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:31.121515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:31.121878 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 12:49:31.122124 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 12:49:31.122575 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":346,"threads_started":1}
I20260924 12:49:31.125303 16733 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45513
I20260924 12:49:31.125691 16733 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:31.126591 16733 heartbeater.cc:511] Master 127.2.81.126:45513 requested a full tablet report, sending...
I20260924 12:49:31.128173 16276 ts_manager.cc:194] Registered new tserver with Master: 196b120126024e6e9d21749756f1484e (127.2.81.67:35969)
I20260924 12:49:31.129079 16276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:53107
I20260924 12:49:31.140210 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:49:31.159842 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.144836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.2.81.126:45513)
0924 12:49:31.145083 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.145090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.145386 (+ 296us) server_negotiation.cc:407] Connection header received
0924 12:49:31.145589 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.145595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.145666 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.145752 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:31.147259 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.148051 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.149604 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.150039 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.150169 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.152735 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:31.152741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:31.152744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:31.152785 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:31.155938 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.157062 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.157067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.157070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.157226 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.157638 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.157642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.157645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.158024 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:31.158610 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.159022 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.159469 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:49:31.165546 16276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:49:31.172573 16276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:49:31.201648 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.187501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:35969 (local address 127.0.0.1:55854)
0924 12:49:31.189100 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:31.189161 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:31.189384 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:31.190669 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:31.190677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:31.190712 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:31.191206 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.191227 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.193967 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.193975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.197813 (+ 3838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.197823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.200429 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.200440 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.200710 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.200775 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 12:49:31.200929 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 12:49:31.201174 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":1,"thread_start_us":2760,"threads_started":1}
I20260924 12:49:31.201705 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.187806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.2.81.67:35969)
0924 12:49:31.190045 (+ 2239us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.190052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.190080 (+ 28us) server_negotiation.cc:407] Connection header received
0924 12:49:31.190159 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.190164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.190376 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.191471 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:31.191807 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.193724 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.198033 (+ 4309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.199885 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.200023 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.200291 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.201100 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.201300 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2073,"thread_start_us":1917,"threads_started":1}
I20260924 12:49:31.204819 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.188573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.2.81.66:35689)
0924 12:49:31.196341 (+ 7768us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.196350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.196381 (+ 31us) server_negotiation.cc:407] Connection header received
0924 12:49:31.196484 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.196492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.196889 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.197270 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:31.198025 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.199471 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.202152 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.203675 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.203822 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.204032 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.204304 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.204492 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7571,"thread_start_us":5702,"threads_started":1}
I20260924 12:49:31.204849 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.187934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35689 (local address 127.0.0.1:54332)
0924 12:49:31.190544 (+ 2610us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:31.190569 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:31.190685 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:31.197327 (+ 6642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:31.197335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:31.197371 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:31.197802 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.197816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.199671 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.199676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.201964 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.201982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.203910 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.203916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.204088 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.204126 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:49:31.204231 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 12:49:31.204406 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2492,"mutex_wait_us":29,"thread_start_us":655,"threads_started":1}
I20260924 12:49:31.205806 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.191278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.2.81.65:33839)
0924 12:49:31.195337 (+ 4059us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.195346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.195379 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:49:31.195510 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.195519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.195883 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.196127 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:31.197579 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.199294 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.202379 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.203786 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.203912 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.204095 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.205321 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.205505 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3848,"thread_start_us":3415,"threads_started":1}
I20260924 12:49:31.206235 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.190724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45072)
0924 12:49:31.193779 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:31.193810 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:31.193977 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:31.196467 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:31.196472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:31.196518 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:31.197241 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.197268 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.199503 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.199509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.202167 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.202183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.204192 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.204199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.204419 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.204452 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:49:31.204532 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 12:49:31.204664 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2907,"mutex_wait_us":108,"thread_start_us":2646,"threads_started":1}
I20260924 12:49:31.207162 16668 tablet_service.cc:1467] Processing CreateTablet for tablet 551be8d3b5b64ae38731d6619c7c31bf (DEFAULT_TABLE table=test-table [id=1b56e875618f4ee790ecc599ad3669a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:31.208454 16668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551be8d3b5b64ae38731d6619c7c31bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:31.208666 16534 tablet_service.cc:1467] Processing CreateTablet for tablet 551be8d3b5b64ae38731d6619c7c31bf (DEFAULT_TABLE table=test-table [id=1b56e875618f4ee790ecc599ad3669a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:31.210156 16534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551be8d3b5b64ae38731d6619c7c31bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:31.209736 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 551be8d3b5b64ae38731d6619c7c31bf (DEFAULT_TABLE table=test-table [id=1b56e875618f4ee790ecc599ad3669a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:31.211056 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551be8d3b5b64ae38731d6619c7c31bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:31.220786 16752 tablet_bootstrap.cc:492] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Bootstrap starting.
I20260924 12:49:31.223891 16752 tablet_bootstrap.cc:654] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:31.224658 16753 tablet_bootstrap.cc:492] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Bootstrap starting.
I20260924 12:49:31.225274 16752 log.cc:824] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:31.226769 16754 tablet_bootstrap.cc:492] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Bootstrap starting.
I20260924 12:49:31.228358 16753 tablet_bootstrap.cc:654] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:31.228677 16752 tablet_bootstrap.cc:492] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: No bootstrap required, opened a new log
I20260924 12:49:31.228992 16752 ts_tablet_manager.cc:1397] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 12:49:31.229864 16753 log.cc:824] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:31.230784 16754 tablet_bootstrap.cc:654] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:31.231973 16754 log.cc:824] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:31.233139 16753 tablet_bootstrap.cc:492] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: No bootstrap required, opened a new log
I20260924 12:49:31.233547 16753 ts_tablet_manager.cc:1397] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 12:49:31.235198 16754 tablet_bootstrap.cc:492] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: No bootstrap required, opened a new log
I20260924 12:49:31.235481 16754 ts_tablet_manager.cc:1397] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 12:49:31.237149 16752 raft_consensus.cc:348] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.237524 16752 raft_consensus.cc:374] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:31.237633 16752 raft_consensus.cc:729] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196b120126024e6e9d21749756f1484e, State: Initialized, Role: FOLLOWER
I20260924 12:49:31.238201 16752 consensus_queue.cc:260] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.239243 16733 heartbeater.cc:503] Master 127.2.81.126:45513 was elected leader, sending a full tablet report...
I20260924 12:49:31.239987 16752 ts_tablet_manager.cc:1428] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 12:49:31.243984 16753 raft_consensus.cc:348] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.244508 16753 raft_consensus.cc:374] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:31.244613 16753 raft_consensus.cc:729] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33fae30b3364a09b4f6bd446833b7f3, State: Initialized, Role: FOLLOWER
I20260924 12:49:31.245329 16753 consensus_queue.cc:260] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.246601 16599 heartbeater.cc:503] Master 127.2.81.126:45513 was elected leader, sending a full tablet report...
I20260924 12:49:31.247466 16753 ts_tablet_manager.cc:1428] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260924 12:49:31.247608 16754 raft_consensus.cc:348] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.248064 16754 raft_consensus.cc:374] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:31.248173 16754 raft_consensus.cc:729] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e62a47082a493090195cfec125a64b, State: Initialized, Role: FOLLOWER
I20260924 12:49:31.248827 16754 consensus_queue.cc:260] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.250535 16754 ts_tablet_manager.cc:1428] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260924 12:49:31.256147 16760 raft_consensus.cc:482] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:49:31.256423 16760 raft_consensus.cc:504] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.258165 16760 leader_election.cc:288] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 196b120126024e6e9d21749756f1484e (127.2.81.67:35969), b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689)
I20260924 12:49:31.265328 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.258583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:46027 (local address 127.2.81.66:35689)
0924 12:49:31.258808 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.258814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.259705 (+ 891us) server_negotiation.cc:407] Connection header received
0924 12:49:31.259778 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.259781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.259837 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.259932 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:31.260552 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.261430 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.263466 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.264247 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.264372 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.264502 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.264912 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.265084 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:49:31.265324 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.258682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:42803 (local address 127.2.81.67:35969)
0924 12:49:31.258883 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.258889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.259576 (+ 687us) server_negotiation.cc:407] Connection header received
0924 12:49:31.259714 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.259720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.259796 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.259871 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:31.260545 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.261419 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.263468 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.264255 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.264375 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.264506 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.264928 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.265084 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 12:49:31.265626 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.258613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:35969 (local address 127.2.81.65:42803)
0924 12:49:31.259452 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:31.259469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:31.259593 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:31.259914 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:31.259918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:31.259950 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:31.260377 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.260392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.261559 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.261564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.263296 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.263310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.264540 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.264547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.264730 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.264776 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 12:49:31.264862 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 12:49:31.265025 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":102,"thread_start_us":475,"threads_started":1}
I20260924 12:49:31.265753 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.258620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35689 (local address 127.2.81.65:46027)
0924 12:49:31.259531 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:31.259549 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:31.259688 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:31.259964 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:31.259969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:31.259992 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:31.260381 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.260391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.261567 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.261572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.263306 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:31.263319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.264506 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.264517 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:31.264717 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.264760 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 12:49:31.264842 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 12:49:31.265015 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":133,"thread_start_us":460,"threads_started":1}
I20260924 12:49:31.266777 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551be8d3b5b64ae38731d6619c7c31bf" candidate_uuid: "59e62a47082a493090195cfec125a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33fae30b3364a09b4f6bd446833b7f3" is_pre_election: true
I20260924 12:49:31.266786 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551be8d3b5b64ae38731d6619c7c31bf" candidate_uuid: "59e62a47082a493090195cfec125a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196b120126024e6e9d21749756f1484e" is_pre_election: true
I20260924 12:49:31.267251 16688 raft_consensus.cc:2454] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e62a47082a493090195cfec125a64b in term 0.
I20260924 12:49:31.267251 16554 raft_consensus.cc:2454] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e62a47082a493090195cfec125a64b in term 0.
I20260924 12:49:31.267989 16356 leader_election.cc:302] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196b120126024e6e9d21749756f1484e, 59e62a47082a493090195cfec125a64b; no voters:
I20260924 12:49:31.268456 16760 raft_consensus.cc:2781] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:49:31.268591 16760 raft_consensus.cc:482] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:49:31.268726 16760 raft_consensus.cc:3037] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:31.271015 16760 raft_consensus.cc:504] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
I20260924 12:49:31.271677 16760 leader_election.cc:288] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [CANDIDATE]: Term 1 election: Requested vote from peers 196b120126024e6e9d21749756f1484e (127.2.81.67:35969), b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689)
I20260924 12:49:31.272251 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551be8d3b5b64ae38731d6619c7c31bf" candidate_uuid: "59e62a47082a493090195cfec125a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33fae30b3364a09b4f6bd446833b7f3"
I20260924 12:49:31.272334 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551be8d3b5b64ae38731d6619c7c31bf" candidate_uuid: "59e62a47082a493090195cfec125a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196b120126024e6e9d21749756f1484e"
I20260924 12:49:31.272472 16554 raft_consensus.cc:3037] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:31.272547 16688 raft_consensus.cc:3037] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:31.275030 16688 raft_consensus.cc:2454] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e62a47082a493090195cfec125a64b in term 1.
I20260924 12:49:31.275050 16554 raft_consensus.cc:2454] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e62a47082a493090195cfec125a64b in term 1.
I20260924 12:49:31.275539 16356 leader_election.cc:302] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 196b120126024e6e9d21749756f1484e, 59e62a47082a493090195cfec125a64b, b33fae30b3364a09b4f6bd446833b7f3; no voters:
I20260924 12:49:31.275916 16760 raft_consensus.cc:2781] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:31.276368 16760 raft_consensus.cc:686] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 1 LEADER]: Becoming Leader. State: Replica: 59e62a47082a493090195cfec125a64b, State: Running, Role: LEADER
I20260924 12:49:31.276839 16760 consensus_queue.cc:237] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } }
W20260924 12:49:31.281862 16466 tablet.cc:2369] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:31.282015 16275 catalog_manager.cc:5495] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e62a47082a493090195cfec125a64b (127.2.81.65). New cstate: current_term: 1 leader_uuid: "59e62a47082a493090195cfec125a64b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } health_report { overall_health: HEALTHY } } }
I20260924 12:49:31.301713 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.295665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.2.81.65:33839)
0924 12:49:31.295930 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.295936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.295956 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:49:31.296055 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.296058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.296138 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.296281 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:49:31.296711 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.297489 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.299151 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.299667 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.299816 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.300488 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:49:31.301079 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:49:31.301219 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.301355 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.301452 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 12:49:31.310712 16688 raft_consensus.cc:1261] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Refusing update from remote peer 59e62a47082a493090195cfec125a64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:31.310711 16554 raft_consensus.cc:1261] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Refusing update from remote peer 59e62a47082a493090195cfec125a64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:31.311928 16763 consensus_queue.cc:1035] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:31.312289 16760 consensus_queue.cc:1035] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:49:31.327528 16600 tablet.cc:2369] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:31.328390 16766 mvcc.cc:205] Tried to move back new op lower bound from 7332881085676797952 to 7332881085554176000. Current Snapshot: MvccSnapshot[applied={T|T < 7332881085676797952}]
I20260924 12:49:31.345356 16275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:31.345664 16275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:31.349699 16275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Sending DeleteTablet for 3 replicas of tablet 551be8d3b5b64ae38731d6619c7c31bf
I20260924 12:49:31.351084 16534 tablet_service.cc:1514] Processing DeleteTablet for tablet 551be8d3b5b64ae38731d6619c7c31bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:31 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:54332
I20260924 12:49:31.351310 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet 551be8d3b5b64ae38731d6619c7c31bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:31 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:45072
I20260924 12:49:31.351512 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16337
I20260924 12:49:31.351872 16668 tablet_service.cc:1514] Processing DeleteTablet for tablet 551be8d3b5b64ae38731d6619c7c31bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:31 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:55854
I20260924 12:49:31.364126 16776 tablet_replica.cc:330] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: stopping tablet replica
I20260924 12:49:31.364915 16776 raft_consensus.cc:2229] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:49:31.365859 16776 raft_consensus.cc:2258] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:31.366776 16777 tablet_replica.cc:330] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: stopping tablet replica
I20260924 12:49:31.366777 16775 tablet_replica.cc:330] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: stopping tablet replica
I20260924 12:49:31.367235 16777 raft_consensus.cc:2229] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:31.367225 16775 raft_consensus.cc:2229] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:31.367743 16777 raft_consensus.cc:2258] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:31.367753 16775 raft_consensus.cc:2258] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:31.370251 16775 ts_tablet_manager.cc:1905] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:31.370258 16777 ts_tablet_manager.cc:1905] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:31.375087 16776 ts_tablet_manager.cc:1905] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:31.377347 16777 ts_tablet_manager.cc:1918] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:31.377545 16777 log.cc:1196] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/551be8d3b5b64ae38731d6619c7c31bf
I20260924 12:49:31.377635 16775 ts_tablet_manager.cc:1918] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:31.377817 16775 log.cc:1196] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/551be8d3b5b64ae38731d6619c7c31bf
I20260924 12:49:31.378250 16777 ts_tablet_manager.cc:1939] T 551be8d3b5b64ae38731d6619c7c31bf P 196b120126024e6e9d21749756f1484e: Deleting consensus metadata
I20260924 12:49:31.378525 16775 ts_tablet_manager.cc:1939] T 551be8d3b5b64ae38731d6619c7c31bf P b33fae30b3364a09b4f6bd446833b7f3: Deleting consensus metadata
I20260924 12:49:31.380371 16263 catalog_manager.cc:4841] TS 196b120126024e6e9d21749756f1484e (127.2.81.67:35969): tablet 551be8d3b5b64ae38731d6619c7c31bf (table test-table [id=1b56e875618f4ee790ecc599ad3669a5]) successfully deleted
I20260924 12:49:31.380805 16261 catalog_manager.cc:4841] TS b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689): tablet 551be8d3b5b64ae38731d6619c7c31bf (table test-table [id=1b56e875618f4ee790ecc599ad3669a5]) successfully deleted
I20260924 12:49:31.383826 16776 ts_tablet_manager.cc:1918] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:31.384042 16776 log.cc:1196] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/551be8d3b5b64ae38731d6619c7c31bf
I20260924 12:49:31.384613 16776 ts_tablet_manager.cc:1939] T 551be8d3b5b64ae38731d6619c7c31bf P 59e62a47082a493090195cfec125a64b: Deleting consensus metadata
I20260924 12:49:31.386691 16263 catalog_manager.cc:4841] TS 59e62a47082a493090195cfec125a64b (127.2.81.65:33839): tablet 551be8d3b5b64ae38731d6619c7c31bf (table test-table [id=1b56e875618f4ee790ecc599ad3669a5]) successfully deleted
Sep 24 12:49:31 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:49:31.398520 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.352599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.2.81.65:33839)
0924 12:49:31.353273 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.353280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.368538 (+ 15258us) server_negotiation.cc:407] Connection header received
0924 12:49:31.380261 (+ 11723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.380268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.380397 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.380499 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:31.382472 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.383570 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.385235 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.385693 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.385822 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.390411 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:31.390424 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:31.390439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:31.390504 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:31.393750 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.394794 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.394802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.394805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.394966 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.395455 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.395461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.395465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.395962 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:31.396388 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.397374 (+ 986us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.398123 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473}
I20260924 12:49:31.399448 16460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:31.502213 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16337
I20260924 12:49:31.517326 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16471
Sep 24 12:49:31 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:49:31.534276 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.518099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.2.81.66:35689)
0924 12:49:31.518346 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.518353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.518373 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:49:31.518468 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.518473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.518542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.518627 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:31.519930 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.520832 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.522330 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.522725 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.522849 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.527719 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:31.527744 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:31.527760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:31.527809 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:31.531190 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.531989 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.531997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.531999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.532141 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.532545 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.532549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.532551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.533047 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:31.533378 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.533700 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.533923 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:49:31.534976 16594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:31.624853 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16471
I20260924 12:49:31.639390 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16605
Sep 24 12:49:31 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:49:31.656188 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:31.640175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.2.81.67:35969)
0924 12:49:31.640369 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:31.640374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:31.640393 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:49:31.640602 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:31.640606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:31.640707 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:31.640797 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:31.642264 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.643112 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.644517 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:31.644945 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:31.645099 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:31.649789 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:31.649807 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:31.649823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:31.649868 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:31.652652 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.653791 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.653800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.653803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.653985 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:31.654440 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:31.654445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:31.654447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:31.654844 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:31.655068 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:31.655648 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 12:49:31.655870 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:49:31.656819 16728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:31.748512 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16605
I20260924 12:49:31.762315 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16244
I20260924 12:49:31.763455 16306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:31.876291 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16244
I20260924 12:49:31.894047 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45513
--webserver_interface=127.2.81.126
--webserver_port=42509
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:32.230145 16783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:32.230453 16783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:32.230573 16783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:32.238006 16783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:32.238129 16783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:32.238175 16783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:32.238253 16783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:32.238356 16783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:32.249665 16783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:32.251044 16783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:32.252394 16783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:32.275043 16790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:32.274998 16789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:32.275076 16792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:32.276753 16783 server_base.cc:1034] running on GCE node
I20260924 12:49:32.277873 16783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:32.279672 16783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:32.280946 16783 hybrid_clock.cc:648] HybridClock initialized: now 1790254172280876 us; error 59 us; skew 500 ppm
I20260924 12:49:32.281476 16783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:32 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254172, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:32.289144 16783 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:32.290966 16783 webserver.cc:460] Webserver started at http://127.2.81.126:42509/ using document root <none> and password file <none>
I20260924 12:49:32.291743 16783 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:32.291917 16783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:32.298439 16783 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:49:32.302901 16799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:32.304863 16783 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:49:32.305104 16783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "ae6eb72f383549f3adbffd8169382c8b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "13b4cdd42556004908ba0070042465a9"
server_key_iv: "a854075f526dfaec9b2485090e69533c"
server_key_version: "encryptionkey@0"
I20260924 12:49:32.306311 16783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:32.319942 16783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:32.324843 16783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:32.325176 16783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:32.344183 16783 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45513
I20260924 12:49:32.344211 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45513 every 8 connection(s)
I20260924 12:49:32.345834 16783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:32.347172 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16783
I20260924 12:49:32.348126 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:33839
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:45513
--webserver_port=45061
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:32.355526 16851 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:49:32.359831 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap starting.
I20260924 12:49:32.380152 16851 log.cc:824] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:32.393378 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:49:32.393919 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap complete.
I20260924 12:49:32.404650 16851 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:32.405644 16851 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6eb72f383549f3adbffd8169382c8b, State: Initialized, Role: FOLLOWER
I20260924 12:49:32.406345 16851 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:32.406598 16851 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:32.406718 16851 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:32.406888 16851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:49:32.411098 16851 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:32.411689 16851 leader_election.cc:302] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6eb72f383549f3adbffd8169382c8b; no voters:
I20260924 12:49:32.412313 16851 leader_election.cc:288] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 12:49:32.412463 16855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:49:32.414702 16851 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:32.419448 16855 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 LEADER]: Becoming Leader. State: Replica: ae6eb72f383549f3adbffd8169382c8b, State: Running, Role: LEADER
I20260924 12:49:32.420118 16855 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:32.422863 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ae6eb72f383549f3adbffd8169382c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } } }
I20260924 12:49:32.423331 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:32.428236 16866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:32.428236 16857 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6eb72f383549f3adbffd8169382c8b. Latest consensus state: current_term: 2 leader_uuid: "ae6eb72f383549f3adbffd8169382c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } } }
I20260924 12:49:32.428591 16857 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:32.437067 16866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:32.438575 16866 catalog_manager.cc:1259] Loaded cluster ID: e3ab8046f39b43439840e3232cf04f00
I20260924 12:49:32.438692 16866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:32.442682 16866 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 12:49:32.460467 16866 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 12:49:32.479393 16874 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 12:49:32.481864 16866 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:32.483616 16866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Loaded TSK: 0
I20260924 12:49:32.485206 16866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 12:49:32.715056 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:32.715389 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:32.715534 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:32.723344 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:32.723496 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:32.723651 16853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:49:32.734251 16853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:33839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=45061
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:32.735971 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:32.737469 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:32.753254 16880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:32.753464 16882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:32.753914 16879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:32.755254 16853 server_base.cc:1034] running on GCE node
I20260924 12:49:32.756083 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:32.757479 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:32.758756 16853 hybrid_clock.cc:648] HybridClock initialized: now 1790254172758653 us; error 57 us; skew 500 ppm
I20260924 12:49:32.759418 16853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:32 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254172, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:32.763836 16853 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:49:32.765856 16853 webserver.cc:460] Webserver started at http://127.2.81.65:45061/ using document root <none> and password file <none>
I20260924 12:49:32.766872 16853 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:32.767068 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:32.774780 16853 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:49:32.778407 16889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:32.780203 16853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:49:32.780440 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "59e62a47082a493090195cfec125a64b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "8868f5248ac4340d38539ead50fbd576"
server_key_iv: "b17b0157af8080e5aeba23a6eab884f0"
server_key_version: "encryptionkey@0"
I20260924 12:49:32.781955 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:32.797070 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:32.801549 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:32.801990 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:32.803146 16853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:32.804661 16853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:32.804773 16853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:32.804873 16853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:32.804945 16853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:32.844657 16853 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:33839
I20260924 12:49:32.844732 17001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:33839 every 8 connection(s)
I20260924 12:49:32.846065 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:49:32.856084 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 16853
I20260924 12:49:32.856855 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:35689
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:45513
--webserver_port=42337
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:49:32 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254172, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:32.870740 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:32.848806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:39403 (local address 127.2.81.126:45513)
0924 12:49:32.849482 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:32.849498 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:32.849870 (+ 372us) server_negotiation.cc:407] Connection header received
0924 12:49:32.851939 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:32.851952 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:32.852295 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:32.852586 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:32.854073 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:32.855349 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:32.857674 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:32.858448 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:32.858599 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:32.862823 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:32.862832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:32.862848 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:32.862891 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:32.865932 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:32.866938 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:32.866945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:32.866950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:32.867124 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:32.867589 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:32.867591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:32.867593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:32.868096 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:32.868248 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:32.869217 (+ 969us) server_negotiation.cc:299] Negotiation successful
0924 12:49:32.869579 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":263,"threads_started":1}
I20260924 12:49:32.870816 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:32.848947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:39403)
0924 12:49:32.849700 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:32.849743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:32.851362 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:32.852701 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:32.852708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:32.853287 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:32.853815 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:32.853834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:32.855543 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:32.855548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:32.857464 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:32.857488 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:32.858681 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:32.858688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:32.858952 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:32.859659 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:32.859687 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:32.862467 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:32.866125 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:32.866136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:32.866155 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:32.866775 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:32.867233 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:32.867244 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:32.867248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:32.867489 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:32.868220 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:32.868228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:32.868644 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 12:49:32.868949 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 12:49:32.869596 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":300,"threads_started":1}
I20260924 12:49:32.874336 17002 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45513
I20260924 12:49:32.874650 17002 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:32.875384 17002 heartbeater.cc:511] Master 127.2.81.126:45513 requested a full tablet report, sending...
I20260924 12:49:32.877900 16816 ts_manager.cc:194] Registered new tserver with Master: 59e62a47082a493090195cfec125a64b (127.2.81.65:33839)
I20260924 12:49:32.880069 16816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:39403
W20260924 12:49:33.194643 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:33.194952 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:33.195053 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:33.203197 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:33.203331 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:33.203454 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:49:33.214509 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=42337
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:33.215884 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:33.217301 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:33.231269 17012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:33.231212 17011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:33.231184 17014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:33.232254 17006 server_base.cc:1034] running on GCE node
I20260924 12:49:33.232940 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:33.234045 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:33.235270 17006 hybrid_clock.cc:648] HybridClock initialized: now 1790254173235229 us; error 42 us; skew 500 ppm
I20260924 12:49:33.235632 17006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:33 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254173, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:33.239584 17006 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:49:33.241904 17006 webserver.cc:460] Webserver started at http://127.2.81.66:42337/ using document root <none> and password file <none>
I20260924 12:49:33.243202 17006 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:33.243433 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:33.250257 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:49:33.253846 17021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:33.255452 17006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:49:33.255638 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "b33fae30b3364a09b4f6bd446833b7f3"
format_stamp: "Formatted at 2026-09-24 12:49:30 on dist-test-slave-9tm1"
server_key: "6e174ee62e2afa48bb063fe0eaa2978d"
server_key_iv: "c6a2758e9d09f5b2bc8ddcea6bab6a93"
server_key_version: "encryptionkey@0"
I20260924 12:49:33.256664 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:33.278548 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:33.283386 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:33.283658 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:33.284689 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:33.286448 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:33.286548 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:33.286660 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:33.286749 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:33.321600 17006 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:35689
I20260924 12:49:33.321657 17133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:35689 every 8 connection(s)
I20260924 12:49:33.323153 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:49:33.333238 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17006
I20260924 12:49:33.333799 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:35969
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:45513
--webserver_port=32901
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:49:33 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254173, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:33.345512 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.325892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:33409 (local address 127.2.81.126:45513)
0924 12:49:33.326098 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.326103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.327291 (+ 1188us) server_negotiation.cc:407] Connection header received
0924 12:49:33.329219 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.329225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.329322 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.329412 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:33.331408 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.332128 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.333859 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.334437 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.334562 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.338569 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:33.338576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:33.338580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:33.338625 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:33.341518 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:33.342590 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:33.342595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:33.342598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:33.342789 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:33.343367 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:33.343372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:33.343375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:33.343801 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:33.343968 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.344843 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.345148 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:49:33.346799 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.326333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:33409)
0924 12:49:33.327081 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.327123 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.328861 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.329702 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.329711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.330511 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:33.331148 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.331171 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.332300 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.332304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.333690 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.333707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.334696 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.334705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.334936 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.335693 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:33.335724 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:33.338267 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:33.341738 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:33.341749 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:33.341770 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:33.342405 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:33.342929 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:33.342934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:33.342938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:33.343192 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:33.343967 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:33.343975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:33.344401 (+ 426us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.344684 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.345256 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260924 12:49:33.348698 17134 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45513
I20260924 12:49:33.349080 17134 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:33.350023 17134 heartbeater.cc:511] Master 127.2.81.126:45513 requested a full tablet report, sending...
I20260924 12:49:33.351981 16816 ts_manager.cc:194] Registered new tserver with Master: b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689)
I20260924 12:49:33.353320 16816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:33409
W20260924 12:49:33.665647 17137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:33.665953 17137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:33.666049 17137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:33.673671 17137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:33.673790 17137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:33.673915 17137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:49:33.683847 17137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:35969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=32901
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:33.685140 17137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:33.686379 17137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:33.699266 17145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:33.699301 17143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:33.699342 17142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:33.699903 17137 server_base.cc:1034] running on GCE node
I20260924 12:49:33.700521 17137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:33.701486 17137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:33.702709 17137 hybrid_clock.cc:648] HybridClock initialized: now 1790254173702647 us; error 48 us; skew 500 ppm
I20260924 12:49:33.703043 17137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:33 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254173, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:33.706079 17137 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:49:33.707535 17137 webserver.cc:460] Webserver started at http://127.2.81.67:32901/ using document root <none> and password file <none>
I20260924 12:49:33.708204 17137 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:33.708317 17137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:33.713896 17137 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:49:33.716706 17152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:33.718108 17137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:49:33.718277 17137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "196b120126024e6e9d21749756f1484e"
format_stamp: "Formatted at 2026-09-24 12:49:31 on dist-test-slave-9tm1"
server_key: "822afad6cf931734f50f79bbeda62e5d"
server_key_iv: "64e040d50e688ea885874c44dc0d0dfa"
server_key_version: "encryptionkey@0"
I20260924 12:49:33.719302 17137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:33.737442 17137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:33.741572 17137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:33.741906 17137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:33.742915 17137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:33.744294 17137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:33.744402 17137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:33.744510 17137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:33.744589 17137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:33.775520 17137 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:35969
I20260924 12:49:33.775556 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:35969 every 8 connection(s)
I20260924 12:49:33.777038 17137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:49:33.778882 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17137
Sep 24 12:49:33 dist-test-slave-9tm1 krb5kdc[16231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254173, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:33.795755 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.779844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.2.81.126:45513)
0924 12:49:33.780010 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.780017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.780515 (+ 498us) server_negotiation.cc:407] Connection header received
0924 12:49:33.780723 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.780734 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.780831 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.780991 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:33.783038 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.783791 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.785720 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.786247 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.786362 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.789734 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:33.789740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:33.789743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:33.789783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:33.792857 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:33.793655 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:33.793659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:33.793661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:33.793815 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:33.794206 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:33.794211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:33.794214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:33.794595 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:33.794844 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.795241 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.795469 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 12:49:33.801440 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.779853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:50841 (local address 127.2.81.126:45513)
0924 12:49:33.781051 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.781056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.781895 (+ 839us) server_negotiation.cc:407] Connection header received
0924 12:49:33.784259 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.784264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.784458 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.784578 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:33.786682 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.787553 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.790055 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.790705 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.790843 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.794889 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:33.794896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:33.794901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:33.794940 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:33.798326 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:33.799220 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:33.799224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:33.799227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:33.799348 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:33.799791 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:33.799794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:33.799796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:33.800089 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:33.800199 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.800893 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.801126 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":1068,"thread_start_us":780,"threads_started":1}
I20260924 12:49:33.802387 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.780468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:50841)
0924 12:49:33.781715 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.781778 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.783845 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.784907 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.784921 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.785765 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:33.786449 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.786480 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.787748 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.787754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.789891 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.789906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.790960 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.790966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.791196 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.792063 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:33.792088 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:33.794566 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:33.798516 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:33.798527 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:33.798545 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:33.799091 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:33.799480 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:33.799484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:33.799487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:33.799698 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:33.800192 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:33.800199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:33.800568 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.800791 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.801246 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":644,"threads_started":1}
I20260924 12:49:33.803989 17265 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45513
I20260924 12:49:33.804291 17265 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:33.805091 17265 heartbeater.cc:511] Master 127.2.81.126:45513 requested a full tablet report, sending...
I20260924 12:49:33.806648 16816 ts_manager.cc:194] Registered new tserver with Master: 196b120126024e6e9d21749756f1484e (127.2.81.67:35969)
I20260924 12:49:33.807493 16816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:50841
I20260924 12:49:33.811419 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:49:33.818639 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.812242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.2.81.126:45513)
0924 12:49:33.812463 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.812470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.812812 (+ 342us) server_negotiation.cc:407] Connection header received
0924 12:49:33.813037 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.813042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.813190 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.813317 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:49:33.813826 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.814557 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.816146 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.816676 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.816792 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.817446 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:49:33.818037 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:49:33.818166 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.818271 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.818368 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:49:33.819711 16816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:49:33.827185 16816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:49:33.851049 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.839619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35689 (local address 127.0.0.1:54360)
0924 12:49:33.841047 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.841065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.841174 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.843165 (+ 1991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.843174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.843205 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:33.843581 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.843593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.845742 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.845747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.848088 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.848108 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.850044 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.850059 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.850292 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.850354 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.850463 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.850672 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"thread_start_us":2874,"threads_started":1}
I20260924 12:49:33.851837 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.838980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:35969 (local address 127.0.0.1:55874)
0924 12:49:33.840106 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.840158 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.840383 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.844246 (+ 3863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.844250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.844268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:33.844772 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.844790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.846951 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.846956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.849222 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.849238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.851041 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.851047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.851208 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.851244 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.851311 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.851458 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":4,"thread_start_us":649,"threads_started":1}
I20260924 12:49:33.852579 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.840175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.2.81.66:35689)
0924 12:49:33.841706 (+ 1531us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.841716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.841754 (+ 38us) server_negotiation.cc:407] Connection header received
0924 12:49:33.841861 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.841870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.842152 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.843820 (+ 1668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:33.844088 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.845583 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.848319 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.849828 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.851709 (+ 1881us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.851911 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.852101 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.852289 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":1192,"threads_started":1}
I20260924 12:49:33.852674 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.839250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.2.81.67:35969)
0924 12:49:33.843700 (+ 4450us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.843707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.843740 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:49:33.843833 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.843841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.844054 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.845035 (+ 981us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:33.845301 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.846788 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.849435 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.850789 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.851938 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.852141 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.852209 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.852353 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4294,"thread_start_us":4152,"threads_started":1}
I20260924 12:49:33.853897 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.840446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45100)
0924 12:49:33.842847 (+ 2401us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.842865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.842976 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.844227 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.844232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.844259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:33.844798 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.844807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.846920 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.846926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.851295 (+ 4369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.851309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.853153 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.853160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.853304 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.853328 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.853415 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.853549 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2270,"mutex_wait_us":48,"thread_start_us":672,"threads_started":1}
I20260924 12:49:33.854386 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.840931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.2.81.65:33839)
0924 12:49:33.843278 (+ 2347us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.843287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.843319 (+ 32us) server_negotiation.cc:407] Connection header received
0924 12:49:33.843431 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.843439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.843685 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.843886 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:33.845020 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.846729 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.851473 (+ 4744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.852775 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.852913 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.853118 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.853485 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.854086 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2199,"thread_start_us":2061,"threads_started":1}
I20260924 12:49:33.856549 17069 tablet_service.cc:1467] Processing CreateTablet for tablet 023d60df5ffd4a51adf76632884d1901 (DEFAULT_TABLE table=test-table [id=ac1a76f5b78246bcb6c627bd1b2d08f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:33.856549 17200 tablet_service.cc:1467] Processing CreateTablet for tablet 023d60df5ffd4a51adf76632884d1901 (DEFAULT_TABLE table=test-table [id=ac1a76f5b78246bcb6c627bd1b2d08f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:33.858057 17069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023d60df5ffd4a51adf76632884d1901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:33.858057 17200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023d60df5ffd4a51adf76632884d1901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:33.857736 16937 tablet_service.cc:1467] Processing CreateTablet for tablet 023d60df5ffd4a51adf76632884d1901 (DEFAULT_TABLE table=test-table [id=ac1a76f5b78246bcb6c627bd1b2d08f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:33.858892 16937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023d60df5ffd4a51adf76632884d1901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:33.874243 17282 tablet_bootstrap.cc:492] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Bootstrap starting.
I20260924 12:49:33.876580 17283 tablet_bootstrap.cc:492] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Bootstrap starting.
I20260924 12:49:33.877686 17281 tablet_bootstrap.cc:492] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Bootstrap starting.
I20260924 12:49:33.879717 17282 tablet_bootstrap.cc:654] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:33.879946 17283 tablet_bootstrap.cc:654] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:33.881162 17281 tablet_bootstrap.cc:654] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:33.881132 17282 log.cc:824] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:33.881132 17283 log.cc:824] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:33.882318 17281 log.cc:824] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:33.883463 17002 heartbeater.cc:503] Master 127.2.81.126:45513 was elected leader, sending a full tablet report...
I20260924 12:49:33.884516 17282 tablet_bootstrap.cc:492] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: No bootstrap required, opened a new log
I20260924 12:49:33.884538 17283 tablet_bootstrap.cc:492] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: No bootstrap required, opened a new log
I20260924 12:49:33.884821 17282 ts_tablet_manager.cc:1397] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 12:49:33.885032 17283 ts_tablet_manager.cc:1397] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 12:49:33.885483 17281 tablet_bootstrap.cc:492] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: No bootstrap required, opened a new log
I20260924 12:49:33.885756 17281 ts_tablet_manager.cc:1397] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260924 12:49:33.893784 17282 raft_consensus.cc:348] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.894202 17282 raft_consensus.cc:374] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:33.894299 17282 raft_consensus.cc:729] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33fae30b3364a09b4f6bd446833b7f3, State: Initialized, Role: FOLLOWER
I20260924 12:49:33.894958 17282 consensus_queue.cc:260] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.896250 17134 heartbeater.cc:503] Master 127.2.81.126:45513 was elected leader, sending a full tablet report...
I20260924 12:49:33.896078 17283 raft_consensus.cc:348] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.896546 17283 raft_consensus.cc:374] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:33.896358 17281 raft_consensus.cc:348] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.896637 17283 raft_consensus.cc:729] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196b120126024e6e9d21749756f1484e, State: Initialized, Role: FOLLOWER
I20260924 12:49:33.896745 17281 raft_consensus.cc:374] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:33.896828 17281 raft_consensus.cc:729] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e62a47082a493090195cfec125a64b, State: Initialized, Role: FOLLOWER
I20260924 12:49:33.896960 17282 ts_tablet_manager.cc:1428] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 12:49:33.897322 17283 consensus_queue.cc:260] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.897713 17281 consensus_queue.cc:260] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.898484 17265 heartbeater.cc:503] Master 127.2.81.126:45513 was elected leader, sending a full tablet report...
I20260924 12:49:33.899139 17283 ts_tablet_manager.cc:1428] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260924 12:49:33.900650 17281 ts_tablet_manager.cc:1428] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260924 12:49:33.920751 17288 raft_consensus.cc:482] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:49:33.921079 17288 raft_consensus.cc:504] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.922931 17288 leader_election.cc:288] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59e62a47082a493090195cfec125a64b (127.2.81.65:33839), b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689)
I20260924 12:49:33.928756 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.923299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35689 (local address 127.2.81.67:54161)
0924 12:49:33.923532 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.923556 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.923706 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.924033 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.924038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.924070 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:33.924297 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.924310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.925490 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.925495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.926821 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.926834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.927951 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.927957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.928139 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.928164 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.928254 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.928397 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 12:49:33.928755 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.923348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:54161 (local address 127.2.81.66:35689)
0924 12:49:33.923549 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.923556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.923637 (+ 81us) server_negotiation.cc:407] Connection header received
0924 12:49:33.923858 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.923862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.923920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.923997 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:33.924443 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.925371 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.926995 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.927789 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.927919 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.928016 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.928306 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.928432 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:49:33.929811 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.923482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:43655 (local address 127.2.81.65:33839)
0924 12:49:33.923705 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.923711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.924515 (+ 804us) server_negotiation.cc:407] Connection header received
0924 12:49:33.924628 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.924633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.924715 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.924810 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:33.925576 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.926319 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.928007 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.928786 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.928922 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.929050 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.929451 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.929591 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 12:49:33.929828 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.923656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.2.81.67:43655)
0924 12:49:33.924409 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:33.924424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:33.924530 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:33.924890 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:33.924894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:33.924918 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:33.925366 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.925381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.926470 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.926475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.927833 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:33.927847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.929024 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.929031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:33.929181 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.929204 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 12:49:33.929283 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 12:49:33.929488 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":452,"threads_started":1}
I20260924 12:49:33.930127 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023d60df5ffd4a51adf76632884d1901" candidate_uuid: "196b120126024e6e9d21749756f1484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33fae30b3364a09b4f6bd446833b7f3" is_pre_election: true
I20260924 12:49:33.930661 17089 raft_consensus.cc:2454] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196b120126024e6e9d21749756f1484e in term 0.
I20260924 12:49:33.931118 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023d60df5ffd4a51adf76632884d1901" candidate_uuid: "196b120126024e6e9d21749756f1484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e62a47082a493090195cfec125a64b" is_pre_election: true
I20260924 12:49:33.931439 17154 leader_election.cc:302] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196b120126024e6e9d21749756f1484e, b33fae30b3364a09b4f6bd446833b7f3; no voters:
I20260924 12:49:33.931654 16957 raft_consensus.cc:2454] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196b120126024e6e9d21749756f1484e in term 0.
I20260924 12:49:33.931860 17288 raft_consensus.cc:2781] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:49:33.932010 17288 raft_consensus.cc:482] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:49:33.932123 17288 raft_consensus.cc:3037] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:33.934475 17288 raft_consensus.cc:504] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.935119 17288 leader_election.cc:288] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [CANDIDATE]: Term 1 election: Requested vote from peers 59e62a47082a493090195cfec125a64b (127.2.81.65:33839), b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689)
I20260924 12:49:33.935582 16957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023d60df5ffd4a51adf76632884d1901" candidate_uuid: "196b120126024e6e9d21749756f1484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e62a47082a493090195cfec125a64b"
I20260924 12:49:33.935698 17089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023d60df5ffd4a51adf76632884d1901" candidate_uuid: "196b120126024e6e9d21749756f1484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33fae30b3364a09b4f6bd446833b7f3"
I20260924 12:49:33.935832 16957 raft_consensus.cc:3037] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:33.935946 17089 raft_consensus.cc:3037] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:33.939132 17089 raft_consensus.cc:2454] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196b120126024e6e9d21749756f1484e in term 1.
I20260924 12:49:33.939136 16957 raft_consensus.cc:2454] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196b120126024e6e9d21749756f1484e in term 1.
I20260924 12:49:33.939684 17156 leader_election.cc:302] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196b120126024e6e9d21749756f1484e, 59e62a47082a493090195cfec125a64b; no voters:
I20260924 12:49:33.940084 17288 raft_consensus.cc:2781] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:33.940549 17288 raft_consensus.cc:686] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 1 LEADER]: Becoming Leader. State: Replica: 196b120126024e6e9d21749756f1484e, State: Running, Role: LEADER
I20260924 12:49:33.941066 17288 consensus_queue.cc:237] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } }
I20260924 12:49:33.945760 16816 catalog_manager.cc:5495] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e reported cstate change: term changed from 0 to 1, leader changed from <none> to 196b120126024e6e9d21749756f1484e (127.2.81.67). New cstate: current_term: 1 leader_uuid: "196b120126024e6e9d21749756f1484e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196b120126024e6e9d21749756f1484e" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 35969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 } health_report { overall_health: UNKNOWN } } }
I20260924 12:49:33.963584 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:33.957874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.2.81.67:35969)
0924 12:49:33.958058 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:33.958062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:33.958081 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:49:33.958244 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:33.958248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:33.958329 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:33.958473 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:49:33.958873 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.959637 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.961101 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:33.961654 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:33.961814 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:33.962498 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:49:33.963031 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:49:33.963153 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:33.963294 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 12:49:33.963379 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 12:49:33.972287 16957 raft_consensus.cc:1261] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Refusing update from remote peer 196b120126024e6e9d21749756f1484e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:33.972287 17089 raft_consensus.cc:1261] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Refusing update from remote peer 196b120126024e6e9d21749756f1484e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:33.973511 17291 consensus_queue.cc:1035] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e62a47082a493090195cfec125a64b" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 33839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:33.973913 17288 consensus_queue.cc:1035] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33fae30b3364a09b4f6bd446833b7f3" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:33.987674 17295 mvcc.cc:205] Tried to move back new op lower bound from 7332881096579555328 to 7332881096466423808. Current Snapshot: MvccSnapshot[applied={T|T < 7332881096579555328}]
I20260924 12:49:34.000924 16816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:34.001221 16816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:34.005071 16816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Sending DeleteTablet for 3 replicas of tablet 023d60df5ffd4a51adf76632884d1901
I20260924 12:49:34.006409 16937 tablet_service.cc:1514] Processing DeleteTablet for tablet 023d60df5ffd4a51adf76632884d1901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:33 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:45100
I20260924 12:49:34.006373 17200 tablet_service.cc:1514] Processing DeleteTablet for tablet 023d60df5ffd4a51adf76632884d1901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:33 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:55874
I20260924 12:49:34.006791 17069 tablet_service.cc:1514] Processing DeleteTablet for tablet 023d60df5ffd4a51adf76632884d1901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:33 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:54360
I20260924 12:49:34.007860 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16853
I20260924 12:49:34.018896 17306 tablet_replica.cc:330] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: stopping tablet replica
I20260924 12:49:34.019464 17306 raft_consensus.cc:2229] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:34.019788 17304 tablet_replica.cc:330] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: stopping tablet replica
I20260924 12:49:34.020044 17306 raft_consensus.cc:2258] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:34.020823 17304 raft_consensus.cc:2229] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:34.021482 17304 raft_consensus.cc:2258] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:34.022773 17305 tablet_replica.cc:330] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: stopping tablet replica
I20260924 12:49:34.023232 17305 raft_consensus.cc:2229] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:49:34.024065 17305 raft_consensus.cc:2258] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:34.024467 17306 ts_tablet_manager.cc:1905] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:34.026630 17305 ts_tablet_manager.cc:1905] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:34.026705 17304 ts_tablet_manager.cc:1905] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:34.031942 17306 ts_tablet_manager.cc:1918] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:34.032137 17306 log.cc:1196] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/023d60df5ffd4a51adf76632884d1901
I20260924 12:49:34.032660 17306 ts_tablet_manager.cc:1939] T 023d60df5ffd4a51adf76632884d1901 P b33fae30b3364a09b4f6bd446833b7f3: Deleting consensus metadata
I20260924 12:49:34.033291 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.009403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.2.81.65:33839)
0924 12:49:34.009597 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:34.009604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:34.009627 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:49:34.009694 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:34.009700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:34.009807 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:34.009924 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:34.011624 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:34.012846 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:34.014816 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:34.015520 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:34.015651 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:34.019342 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:34.019367 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:34.019395 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:34.019440 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:34.024763 (+ 5323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:34.026988 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:34.026997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:34.027000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:34.027176 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:34.027664 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:34.027669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:34.027672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:34.028161 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:34.028484 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:34.032460 (+ 3976us) server_negotiation.cc:299] Negotiation successful
0924 12:49:34.032835 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 12:49:34.034412 16997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:34.034621 16801 catalog_manager.cc:4841] TS b33fae30b3364a09b4f6bd446833b7f3 (127.2.81.66:35689): tablet 023d60df5ffd4a51adf76632884d1901 (table test-table [id=ac1a76f5b78246bcb6c627bd1b2d08f9]) successfully deleted
I20260924 12:49:34.037856 17305 ts_tablet_manager.cc:1918] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:34.038210 17305 log.cc:1196] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/023d60df5ffd4a51adf76632884d1901
I20260924 12:49:34.039217 17305 ts_tablet_manager.cc:1939] T 023d60df5ffd4a51adf76632884d1901 P 196b120126024e6e9d21749756f1484e: Deleting consensus metadata
I20260924 12:49:34.041333 16803 catalog_manager.cc:4841] TS 196b120126024e6e9d21749756f1484e (127.2.81.67:35969): tablet 023d60df5ffd4a51adf76632884d1901 (table test-table [id=ac1a76f5b78246bcb6c627bd1b2d08f9]) successfully deleted
I20260924 12:49:34.126746 17304 ts_tablet_manager.cc:1918] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:34.127023 17304 log.cc:1196] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/023d60df5ffd4a51adf76632884d1901
I20260924 12:49:34.127404 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16853
I20260924 12:49:34.141968 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17006
W20260924 12:49:34.142055 16803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:33839 (error 108)
I20260924 12:49:34.144102 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.143645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45108)
0924 12:49:34.143812 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.143924 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 12:49:34.145402 16803 catalog_manager.cc:4584] TS 59e62a47082a493090195cfec125a64b (127.2.81.65:33839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 023d60df5ffd4a51adf76632884d1901: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
I20260924 12:49:34.159114 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.143391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.2.81.66:35689)
0924 12:49:34.143568 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:34.143574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:34.143594 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:49:34.143658 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:34.143663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:34.143764 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:34.143866 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:34.145324 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:34.146496 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:34.148162 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:34.148836 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:34.149014 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:34.152376 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:34.152392 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:34.152411 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:34.152455 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:34.156049 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:34.156884 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:34.156892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:34.156895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:34.157091 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:34.157485 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:34.157491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:34.157494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:34.157905 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:34.158199 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:34.158535 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 12:49:34.158768 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 12:49:34.159930 17129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:34.198977 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.198490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45122)
0924 12:49:34.198697 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.198791 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 12:49:34.200062 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.199700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45132)
0924 12:49:34.199846 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.199939 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 12:49:34.261252 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.260635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45134)
0924 12:49:34.260899 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.261045 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 12:49:34.262495 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.262068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45146)
0924 12:49:34.262239 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.262336 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 12:49:34.285621 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17006
I20260924 12:49:34.300411 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17137
I20260924 12:49:34.317984 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.301189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.2.81.67:35969)
0924 12:49:34.301350 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:34.301354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:34.301591 (+ 237us) server_negotiation.cc:407] Connection header received
0924 12:49:34.301740 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:34.301747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:34.301841 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:34.301957 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:34.303325 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:34.304682 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:34.306248 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:34.306944 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:34.307122 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:34.310054 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:34.310072 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:34.310093 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:34.310155 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:34.314493 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:34.315466 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:34.315476 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:34.315479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:34.315682 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:34.316129 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:34.316135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:34.316138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:34.316623 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:34.316947 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:34.317313 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 12:49:34.317572 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 12:49:34.318723 17260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:34.343434 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.342828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45150)
0924 12:49:34.343075 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.343189 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 12:49:34.344650 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:34.344201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:33839 (local address 127.0.0.1:45158)
0924 12:49:34.344426 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:34.344514 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 12:49:34.442092 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17137
I20260924 12:49:34.456741 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 16783
I20260924 12:49:34.457867 16846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:34.571762 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 16783
I20260924 12:49:34.590328 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45513
--webserver_interface=127.2.81.126
--webserver_port=42509
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:34.928411 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:34.928718 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:34.928838 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:34.936290 17312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:34.936426 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:34.936478 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:34.936558 17312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:34.936615 17312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:34.947029 17312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:34.948372 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:34.949770 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:34.962225 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:34.962301 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:34.962271 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:34.963563 17312 server_base.cc:1034] running on GCE node
I20260924 12:49:34.964450 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:34.966074 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:34.967334 17312 hybrid_clock.cc:648] HybridClock initialized: now 1790254174967262 us; error 59 us; skew 500 ppm
I20260924 12:49:34.967726 17312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:34 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254174, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:34.971091 17312 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:34.972724 17312 webserver.cc:460] Webserver started at http://127.2.81.126:42509/ using document root <none> and password file <none>
I20260924 12:49:34.973512 17312 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:34.973639 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:34.979244 17312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:49:34.982208 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:34.983579 17312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:49:34.983748 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "ae6eb72f383549f3adbffd8169382c8b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "13b4cdd42556004908ba0070042465a9"
server_key_iv: "a854075f526dfaec9b2485090e69533c"
server_key_version: "encryptionkey@0"
I20260924 12:49:34.984719 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:35.014519 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:35.018803 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:35.019090 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:35.038575 17312 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45513
I20260924 12:49:35.038611 17378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45513 every 8 connection(s)
I20260924 12:49:35.040179 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:35.044401 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17312
I20260924 12:49:35.045264 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:33839
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:45513
--webserver_port=45061
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:35.048146 17379 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:49:35.051739 17379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap starting.
I20260924 12:49:35.082365 17379 log.cc:824] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:35.105448 17379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:49:35.106204 17379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b: Bootstrap complete.
I20260924 12:49:35.120173 17379 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:35.121481 17379 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6eb72f383549f3adbffd8169382c8b, State: Initialized, Role: FOLLOWER
I20260924 12:49:35.122232 17379 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:35.122524 17379 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:35.122648 17379 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:35.122846 17379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:49:35.128893 17379 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:35.129658 17379 leader_election.cc:302] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6eb72f383549f3adbffd8169382c8b; no voters:
I20260924 12:49:35.130403 17379 leader_election.cc:288] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 12:49:35.130548 17383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 3 FOLLOWER]: Leader election won for term 3
I20260924 12:49:35.131639 17383 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [term 3 LEADER]: Becoming Leader. State: Replica: ae6eb72f383549f3adbffd8169382c8b, State: Running, Role: LEADER
I20260924 12:49:35.132298 17383 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } }
I20260924 12:49:35.132757 17379 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:35.134809 17385 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6eb72f383549f3adbffd8169382c8b. Latest consensus state: current_term: 3 leader_uuid: "ae6eb72f383549f3adbffd8169382c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } } }
I20260924 12:49:35.135313 17385 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:35.134814 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ae6eb72f383549f3adbffd8169382c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6eb72f383549f3adbffd8169382c8b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45513 } } }
I20260924 12:49:35.135772 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6eb72f383549f3adbffd8169382c8b [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:35.136752 17389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:35.159276 17389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:35.161206 17389 catalog_manager.cc:1259] Loaded cluster ID: e3ab8046f39b43439840e3232cf04f00
I20260924 12:49:35.161559 17389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 12:49:35.163908 17389 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbf2b82a9ad google::LogMessage::Fail() at ??:0
@ 0x7fbf2b82d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbf2b82a4de google::LogMessage::Flush() at ??:0
@ 0x7fbf2b82ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbf41261512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbf4125e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbf412e6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbf412e6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf2e476345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbf2e47991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbf2e479799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbf2e454eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbf3172e6db start_thread at ??:0
@ 0x7fbf2a6c461f clone at ??:0
W20260924 12:49:35.475080 17381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:35.475425 17381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:35.475553 17381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:35.483095 17381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:35.483213 17381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:35.483376 17381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:49:35.493640 17381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:33839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=45061
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:35.494935 17381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:35.496204 17381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:35.509109 17420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:35.509184 17423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:35.509145 17421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:35.509725 17381 server_base.cc:1034] running on GCE node
I20260924 12:49:35.510474 17381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:35.511423 17381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:35.512652 17381 hybrid_clock.cc:648] HybridClock initialized: now 1790254175512603 us; error 39 us; skew 500 ppm
I20260924 12:49:35.513001 17381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:35 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:35.516150 17381 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:49:35.518007 17381 webserver.cc:460] Webserver started at http://127.2.81.65:45061/ using document root <none> and password file <none>
I20260924 12:49:35.518713 17381 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:35.518836 17381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:35.524564 17381 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:49:35.527429 17430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:35.528841 17381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 12:49:35.529066 17381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "59e62a47082a493090195cfec125a64b"
format_stamp: "Formatted at 2026-09-24 12:49:29 on dist-test-slave-9tm1"
server_key: "8868f5248ac4340d38539ead50fbd576"
server_key_iv: "b17b0157af8080e5aeba23a6eab884f0"
server_key_version: "encryptionkey@0"
I20260924 12:49:35.530108 17381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:35.549361 17381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:35.554278 17381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:35.554546 17381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:35.555526 17381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:35.557353 17437 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260924 12:49:35.562681 17437 ts_tablet_manager.cc:1852] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 12:49:35.563669 17437 ts_tablet_manager.cc:1905] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:35.571012 17437 ts_tablet_manager.cc:1918] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:35.571252 17437 ts_tablet_manager.cc:1939] T 023d60df5ffd4a51adf76632884d1901 P 59e62a47082a493090195cfec125a64b: Deleting consensus metadata
I20260924 12:49:35.571592 17381 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 12:49:35.571714 17381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260924 12:49:35.571834 17381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:35.571913 17381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:35.602943 17381 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:33839
I20260924 12:49:35.602978 17543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:33839 every 8 connection(s)
I20260924 12:49:35.604164 17381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:49:35.608839 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:35.607054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:44111)
0924 12:49:35.607788 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:35.608010 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":308,"threads_started":1}
W20260924 12:49:35.609409 17544 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45513 (0 consecutive failures): Network error: Failed to ping master at 127.2.81.126:45513: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
I20260924 12:49:35.610146 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:35.609860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:42587)
0924 12:49:35.609991 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:35.610053 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 12:49:35.610857 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:35.610584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:38785)
0924 12:49:35.610684 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:35.610752 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 12:49:35.611052 17544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 12:49:35.613525 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17381
I20260924 12:49:35.614094 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:35689
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:45513
--webserver_port=42337
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:35.937539 17547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:35.937831 17547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:35.937952 17547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:35.945443 17547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:35.945559 17547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:35.945709 17547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:49:35.955940 17547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=42337
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:35.957311 17547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:35.958597 17547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:35.971638 17552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:35.971611 17555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:35.971678 17553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:35.972318 17547 server_base.cc:1034] running on GCE node
I20260924 12:49:35.972960 17547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:35.973975 17547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:35.975176 17547 hybrid_clock.cc:648] HybridClock initialized: now 1790254175975120 us; error 47 us; skew 500 ppm
I20260924 12:49:35.975519 17547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:35 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:35.978565 17547 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:49:35.980150 17547 webserver.cc:460] Webserver started at http://127.2.81.66:42337/ using document root <none> and password file <none>
I20260924 12:49:35.980844 17547 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:35.980949 17547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:35.986838 17547 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:49:35.991562 17562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:35.993054 17547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:49:35.993232 17547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "b33fae30b3364a09b4f6bd446833b7f3"
format_stamp: "Formatted at 2026-09-24 12:49:30 on dist-test-slave-9tm1"
server_key: "6e174ee62e2afa48bb063fe0eaa2978d"
server_key_iv: "c6a2758e9d09f5b2bc8ddcea6bab6a93"
server_key_version: "encryptionkey@0"
I20260924 12:49:35.994309 17547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:36.018707 17547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:36.023299 17547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:36.023607 17547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:36.024696 17547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:36.026257 17547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:36.026360 17547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:36.026484 17547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:36.026556 17547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:36.060074 17547 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:35689
I20260924 12:49:36.060133 17674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:35689 every 8 connection(s)
I20260924 12:49:36.061594 17547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:49:36.066619 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17547
I20260924 12:49:36.067386 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:35969
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:45513
--webserver_port=32901
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:49:36.067796 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.064676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:44501)
0924 12:49:36.066053 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.066446 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":526,"threads_started":1}
W20260924 12:49:36.068653 17675 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45513 (0 consecutive failures): Network error: Failed to ping master at 127.2.81.126:45513: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
I20260924 12:49:36.069725 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.069270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:48647)
0924 12:49:36.069465 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.069565 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 12:49:36.070719 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.070319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:54975)
0924 12:49:36.070476 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.070574 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 12:49:36.071043 17675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 12:49:36.394667 17678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:36.394982 17678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:36.395117 17678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:36.402509 17678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:36.402638 17678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:36.402794 17678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:49:36.413141 17678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:35969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=32901
--tserver_master_addrs=127.2.81.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:36.414971 17678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:36.416603 17678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:36.429512 17683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:36.429476 17686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:36.429540 17684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:36.430371 17678 server_base.cc:1034] running on GCE node
I20260924 12:49:36.430968 17678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:36.431903 17678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:36.433115 17678 hybrid_clock.cc:648] HybridClock initialized: now 1790254176433044 us; error 53 us; skew 500 ppm
I20260924 12:49:36.433501 17678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:36 dist-test-slave-9tm1 krb5kdc[16231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254176, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:36.436896 17678 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:49:36.438539 17678 webserver.cc:460] Webserver started at http://127.2.81.67:32901/ using document root <none> and password file <none>
I20260924 12:49:36.439231 17678 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:36.439337 17678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:36.445140 17678 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 12:49:36.448128 17693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:36.449947 17678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:49:36.450133 17678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "196b120126024e6e9d21749756f1484e"
format_stamp: "Formatted at 2026-09-24 12:49:31 on dist-test-slave-9tm1"
server_key: "822afad6cf931734f50f79bbeda62e5d"
server_key_iv: "64e040d50e688ea885874c44dc0d0dfa"
server_key_version: "encryptionkey@0"
I20260924 12:49:36.451164 17678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:36.465934 17678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:36.470752 17678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:36.471030 17678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:36.472082 17678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:36.473590 17678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:36.473690 17678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:36.473798 17678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:36.473869 17678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:36.503901 17678 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:35969
I20260924 12:49:36.503959 17805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:35969 every 8 connection(s)
I20260924 12:49:36.505438 17678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:49:36.510331 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.508346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:34627)
0924 12:49:36.509202 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.509438 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":336,"threads_started":1}
I20260924 12:49:36.510353 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17678
W20260924 12:49:36.511094 17806 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45513 (0 consecutive failures): Network error: Failed to ping master at 127.2.81.126:45513: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
I20260924 12:49:36.512009 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.511611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:51647)
0924 12:49:36.511777 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.511862 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 12:49:36.512871 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.512567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:58591)
0924 12:49:36.512689 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.512763 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 12:49:36.513173 17806 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 12:49:36.612514 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:36.611703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:56453)
0924 12:49:36.612174 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:36.612280 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260924 12:49:37.072743 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:37.071859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:51761)
0924 12:49:37.072393 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:37.072502 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":271,"threads_started":1}
I20260924 12:49:37.514518 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:37.513725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:38937)
0924 12:49:37.514190 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:37.514298 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260924 12:49:37.614279 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:37.613558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:49285)
0924 12:49:37.614015 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:37.614106 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260924 12:49:38.074636 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:38.073827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:45195)
0924 12:49:38.074351 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:38.074447 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":252,"threads_started":1}
I20260924 12:49:38.516242 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:38.515506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:48829)
0924 12:49:38.515985 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:38.516071 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":234,"threads_started":1}
I20260924 12:49:38.616114 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:38.615306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:38095)
0924 12:49:38.615852 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:38.615950 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":291,"threads_started":1}
I20260924 12:49:39.076537 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:39.075735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:46007)
0924 12:49:39.076245 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:39.076338 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":282,"threads_started":1}
I20260924 12:49:39.517961 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:39.517188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:50401)
0924 12:49:39.517695 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:39.517785 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":247,"threads_started":1}
I20260924 12:49:39.618136 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:39.617256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:60411)
0924 12:49:39.617835 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:39.617927 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":264,"threads_started":1}
I20260924 12:49:40.078431 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:40.077638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:36121)
0924 12:49:40.078147 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:40.078249 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":233,"threads_started":1}
I20260924 12:49:40.519738 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:40.518966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:44079)
0924 12:49:40.519422 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:40.519537 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260924 12:49:40.619845 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:40.619134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:41445)
0924 12:49:40.619602 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:40.619687 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":207,"threads_started":1}
I20260924 12:49:41.080142 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:41.079386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:58145)
0924 12:49:41.079865 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:41.079951 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260924 12:49:41.521375 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:41.520629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:37595)
0924 12:49:41.521121 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:41.521200 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":236,"threads_started":1}
I20260924 12:49:41.621686 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:41.620866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:37185)
0924 12:49:41.621377 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:41.621496 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":297,"threads_started":1}
I20260924 12:49:42.082135 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:42.081353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:34921)
0924 12:49:42.081864 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:42.081966 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
I20260924 12:49:42.523051 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:42.522256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:44121)
0924 12:49:42.522788 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:42.522881 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":307,"threads_started":1}
I20260924 12:49:42.623571 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:42.622708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:49353)
0924 12:49:42.623240 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:42.623327 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":260,"threads_started":1}
I20260924 12:49:43.084169 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:43.083319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:51501)
0924 12:49:43.083861 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:43.083957 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":262,"threads_started":1}
I20260924 12:49:43.524788 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:43.524028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:34001)
0924 12:49:43.524490 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:43.524573 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":206,"threads_started":1}
I20260924 12:49:43.625416 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:43.624625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:52671)
0924 12:49:43.625137 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:43.625230 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":282,"threads_started":1}
I20260924 12:49:44.086220 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:44.085433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:33295)
0924 12:49:44.085968 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:44.086060 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":287,"threads_started":1}
I20260924 12:49:44.526531 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:44.525792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:40755)
0924 12:49:44.526275 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:44.526364 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260924 12:49:44.627192 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:44.626415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:44809)
0924 12:49:44.626937 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:44.627037 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260924 12:49:45.088111 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:45.087235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:36217)
0924 12:49:45.087802 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:45.087906 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":288,"threads_started":1}
I20260924 12:49:45.528235 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:45.527477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:48827)
0924 12:49:45.527977 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:45.528066 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":235,"threads_started":1}
I20260924 12:49:45.628922 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:45.628206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:47249)
0924 12:49:45.628649 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:45.628736 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
I20260924 12:49:46.089959 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:46.089234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:60859)
0924 12:49:46.089701 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:46.089786 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":257,"threads_started":1}
I20260924 12:49:46.530015 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:46.529251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:49589)
0924 12:49:46.529761 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:46.529848 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":278,"threads_started":1}
I20260924 12:49:46.630633 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:46.629852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:45603)
0924 12:49:46.630358 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:46.630457 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":273,"threads_started":1}
I20260924 12:49:47.091615 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:47.090881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:35151)
0924 12:49:47.091316 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:47.091429 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260924 12:49:47.531688 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:47.530973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:59337)
0924 12:49:47.531398 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:47.531481 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":198,"threads_started":1}
I20260924 12:49:47.632390 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:47.631655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:51029)
0924 12:49:47.632127 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:47.632217 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":246,"threads_started":1}
I20260924 12:49:48.093461 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:48.092646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:48453)
0924 12:49:48.093171 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:48.093264 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":241,"threads_started":1}
I20260924 12:49:48.533339 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:48.532614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:41563)
0924 12:49:48.533061 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:48.533146 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":200,"threads_started":1}
I20260924 12:49:48.634305 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:48.633528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:37485)
0924 12:49:48.634022 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:48.634131 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":224,"threads_started":1}
I20260924 12:49:49.095386 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:49.094563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:38471)
0924 12:49:49.095081 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:49.095196 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":258,"threads_started":1}
I20260924 12:49:49.535090 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:49.534297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:34981)
0924 12:49:49.534791 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:49.534886 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":243,"threads_started":1}
I20260924 12:49:49.635989 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:49.635234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:47695)
0924 12:49:49.635687 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:49.635776 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260924 12:49:50.097093 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:50.096324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:52283)
0924 12:49:50.096768 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:50.096850 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260924 12:49:50.536633 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:50.535940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:40699)
0924 12:49:50.536362 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:50.536446 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":193,"threads_started":1}
I20260924 12:49:50.637923 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:50.637211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.65:36815)
0924 12:49:50.637666 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:50.637755 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260924 12:49:51.099191 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:51.098302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.66:41113)
0924 12:49:51.098797 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:51.098948 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260924 12:49:51.517596 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17381
I20260924 12:49:51.538621 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:51.537795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45513 (local address 127.2.81.67:42557)
0924 12:49:51.538301 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:51.538452 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
I20260924 12:49:51.541018 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:51.518892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.2.81.65:33839)
0924 12:49:51.519453 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:51.519467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:51.519533 (+ 66us) server_negotiation.cc:407] Connection header received
0924 12:49:51.521298 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:51.521311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:51.521760 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:51.522056 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:51.523423 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:51.524661 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:51.526048 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:51.526746 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:51.526889 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:51.534270 (+ 7381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:51.534280 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:51.534299 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:51.534347 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:51.537467 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:51.538395 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:51.538403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:51.538412 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:51.538624 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:51.539039 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:51.539043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:51.539045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:51.539666 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:51.539908 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:51.540276 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 12:49:51.540690 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":301,"threads_started":1}
I20260924 12:49:51.542387 17539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:51.627743 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17381
I20260924 12:49:51.641300 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17547
I20260924 12:49:51.660420 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:51.642274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.2.81.66:35689)
0924 12:49:51.642795 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:51.642809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:51.642846 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:49:51.644375 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:51.644385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:51.644829 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:51.645182 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:51.646611 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:51.647943 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:51.649529 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:51.650271 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:51.650440 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:51.653197 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:51.653211 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:51.653235 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:51.653302 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:51.656875 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:51.657726 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:51.657734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:51.657741 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:51.657950 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:51.658331 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:51.658335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:51.658337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:51.659061 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:51.659324 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:51.659744 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 12:49:51.660127 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260924 12:49:51.661918 17670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:51.745398 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17547
I20260924 12:49:51.759393 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17678
I20260924 12:49:51.779279 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:51.760268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.2.81.67:35969)
0924 12:49:51.760805 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:51.760819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:51.760860 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:49:51.762791 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:51.762803 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:51.763246 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:51.763583 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:51.765103 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:51.766656 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:51.768225 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:51.769097 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:51.769277 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:51.772146 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:51.772157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:51.772175 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:51.772239 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:51.775600 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:51.776467 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:51.776475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:51.776483 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:51.776683 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:51.777158 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:51.777162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:51.777164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:51.777745 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:51.778038 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:51.778517 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 12:49:51.778969 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":231,"threads_started":1}
I20260924 12:49:51.780751 17801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:51.863664 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17678
2026-09-24T12:49:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24278 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:49:51 dist-test-slave-9tm1 krb5kdc[17863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:49:51 dist-test-slave-9tm1 krb5kdc[17863](info): set up 2 sockets
Sep 24 12:49:51 dist-test-slave-9tm1 krb5kdc[17863](info): commencing operation
krb5kdc: starting...
W20260924 12:49:52.909860 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:49:53 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:49:53.164912 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45209
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:53.515081 17876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:53.515401 17876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:53.515534 17876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:53.522977 17876 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 12:49:53.523103 17876 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 12:49:53.523157 17876 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 12:49:53.523226 17876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:49:53.523294 17876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:53.523394 17876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:53.523451 17876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:49:53.523523 17876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:49:53.533856 17876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:53.535157 17876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:53.536481 17876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:53.548362 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:53.548363 17884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:53.548393 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:53.549191 17876 server_base.cc:1034] running on GCE node
I20260924 12:49:53.550012 17876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:53.551471 17876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:53.552724 17876 hybrid_clock.cc:648] HybridClock initialized: now 1790254193552650 us; error 51 us; skew 500 ppm
I20260924 12:49:53.553120 17876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:53 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:53.557627 17876 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:49:53.559386 17876 webserver.cc:460] Webserver started at http://127.2.81.126:39341/ using document root <none> and password file <none>
I20260924 12:49:53.560143 17876 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:53.560245 17876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:53.560518 17876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:53.563694 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "b999ae8319154608b62dffa05c9d82f9"
format_stamp: "Formatted at 2026-09-24 12:49:53 on dist-test-slave-9tm1"
server_key: "12ac9b933c451b2bbdb09efb59342997"
server_key_iv: "e817d4a5d20483e99ed774388697e723"
server_key_version: "encryptionkey@0"
I20260924 12:49:53.564483 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "b999ae8319154608b62dffa05c9d82f9"
format_stamp: "Formatted at 2026-09-24 12:49:53 on dist-test-slave-9tm1"
server_key: "12ac9b933c451b2bbdb09efb59342997"
server_key_iv: "e817d4a5d20483e99ed774388697e723"
server_key_version: "encryptionkey@0"
I20260924 12:49:53.570088 17876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 12:49:53.574222 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:53.575790 17876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:49:53.575996 17876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "b999ae8319154608b62dffa05c9d82f9"
format_stamp: "Formatted at 2026-09-24 12:49:53 on dist-test-slave-9tm1"
server_key: "12ac9b933c451b2bbdb09efb59342997"
server_key_iv: "e817d4a5d20483e99ed774388697e723"
server_key_version: "encryptionkey@0"
I20260924 12:49:53.576231 17876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:53.592360 17876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:53.597445 17876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:53.597797 17876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:53.616331 17876 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45209
I20260924 12:49:53.616351 17942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45209 every 8 connection(s)
I20260924 12:49:53.617730 17876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:49:53.618419 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17876
I20260924 12:49:53.618620 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:49:53.619453 2373 external_mini_cluster.cc:1428] Setting key 3886b1b9166f3101979ab4d1731e03bd
I20260924 12:49:53.622409 17943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:53.633626 17943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap starting.
Sep 24 12:49:53 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:53.637212 17943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:53.638375 17943 log.cc:824] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:53.641412 17943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: No bootstrap required, opened a new log
I20260924 12:49:53.642756 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:53.621623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.2.81.126:45209)
0924 12:49:53.622473 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:53.622490 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:53.622545 (+ 55us) server_negotiation.cc:407] Connection header received
0924 12:49:53.624198 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:53.624220 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:53.624746 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:53.625183 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:53.626686 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:53.628113 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:53.629578 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:53.630347 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:53.630552 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:53.635009 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:53.635019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:53.635035 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:53.635084 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:53.638251 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:53.639118 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:53.639126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:53.639132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:53.639331 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:53.639717 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:53.639721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:53.639723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:53.640387 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:53.640697 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:53.641210 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 12:49:53.641690 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":345,"threads_started":1}
I20260924 12:49:53.650413 17943 raft_consensus.cc:348] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:49:53.650862 17943 raft_consensus.cc:374] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:53.650962 17943 raft_consensus.cc:729] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b999ae8319154608b62dffa05c9d82f9, State: Initialized, Role: FOLLOWER
I20260924 12:49:53.651561 17943 consensus_queue.cc:260] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:49:53.652012 17943 raft_consensus.cc:388] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:49:53.652275 17943 raft_consensus.cc:482] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:49:53.652450 17943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:53.655983 17943 raft_consensus.cc:504] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:49:53.656569 17943 leader_election.cc:302] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b999ae8319154608b62dffa05c9d82f9; no voters:
I20260924 12:49:53.657209 17943 leader_election.cc:288] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:49:53.657342 17948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:53.658005 17948 raft_consensus.cc:686] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 LEADER]: Becoming Leader. State: Replica: b999ae8319154608b62dffa05c9d82f9, State: Running, Role: LEADER
I20260924 12:49:53.658540 17948 consensus_queue.cc:237] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:49:53.659224 17943 sys_catalog.cc:564] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:49:53.661367 17949 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b999ae8319154608b62dffa05c9d82f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } } }
I20260924 12:49:53.661774 17949 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:53.662490 17950 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b999ae8319154608b62dffa05c9d82f9. Latest consensus state: current_term: 1 leader_uuid: "b999ae8319154608b62dffa05c9d82f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } } }
I20260924 12:49:53.662851 17950 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:49:53.663707 17953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:49:53.669126 17953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:49:53.679288 17953 catalog_manager.cc:1347] Generated new cluster ID: 5f2252a736e94ae084748f63738e0e50
I20260924 12:49:53.679436 17953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:49:53.712697 17953 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:49:53.714633 17953 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:49:53.722218 17953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Generated new TSK 0
I20260924 12:49:53.723327 17953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:49:53.789031 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45209
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:54.125464 17969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:54.125761 17969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:54.125882 17969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:54.133320 17969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:54.133466 17969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:54.133611 17969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:49:54.143677 17969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:54.145020 17969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:54.146317 17969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:54.159435 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:54.159540 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:54.159471 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:54.160192 17969 server_base.cc:1034] running on GCE node
I20260924 12:49:54.160794 17969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:54.161759 17969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:54.162978 17969 hybrid_clock.cc:648] HybridClock initialized: now 1790254194162928 us; error 36 us; skew 500 ppm
I20260924 12:49:54.163357 17969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:54 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:54.166643 17969 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:49:54.168049 17969 webserver.cc:460] Webserver started at http://127.2.81.65:36031/ using document root <none> and password file <none>
I20260924 12:49:54.168686 17969 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:54.168784 17969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:54.169091 17969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:54.171906 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "f9be7d7f497c4d118192c20177a5c455"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "295fb04499c834161d0a6a56f37f9db2"
server_key_iv: "659b38f6d14e11b0b38120421e3b3858"
server_key_version: "encryptionkey@0"
I20260924 12:49:54.172603 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "f9be7d7f497c4d118192c20177a5c455"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "295fb04499c834161d0a6a56f37f9db2"
server_key_iv: "659b38f6d14e11b0b38120421e3b3858"
server_key_version: "encryptionkey@0"
I20260924 12:49:54.177866 17969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 12:49:54.181572 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:54.182947 17969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 12:49:54.183107 17969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "f9be7d7f497c4d118192c20177a5c455"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "295fb04499c834161d0a6a56f37f9db2"
server_key_iv: "659b38f6d14e11b0b38120421e3b3858"
server_key_version: "encryptionkey@0"
I20260924 12:49:54.183310 17969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:54.201610 17969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:54.206357 17969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:54.206661 17969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:54.207849 17969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:54.209342 17969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:54.209420 17969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:54.209538 17969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:54.209597 17969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:54.241350 17969 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:45069
I20260924 12:49:54.241443 18096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:45069 every 8 connection(s)
I20260924 12:49:54.242664 17969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:49:54.252669 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 17969
I20260924 12:49:54.252861 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:49:54.253679 2373 external_mini_cluster.cc:1428] Setting key 03759a6eb3e21e3c3720407cd955b798
Sep 24 12:49:54 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:54.263434 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:54.245117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:38611 (local address 127.2.81.126:45209)
0924 12:49:54.245554 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:54.245559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:54.246546 (+ 987us) server_negotiation.cc:407] Connection header received
0924 12:49:54.248388 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:54.248394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:54.248478 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:54.248577 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:54.250161 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.250866 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.252491 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.253023 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.253172 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:54.256956 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:54.256963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:54.256968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:54.257036 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:54.260110 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:54.261014 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:54.261019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:54.261023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:54.261169 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:54.261664 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:54.261668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:54.261671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:54.262014 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:54.262150 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:54.262876 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 12:49:54.263103 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":214,"threads_started":1}
I20260924 12:49:54.264354 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:54.245676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:38611)
0924 12:49:54.246393 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:54.246430 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:54.248102 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:54.248822 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:54.248831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:54.249407 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:54.249984 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:54.250002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.251021 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.251025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:54.252314 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:54.252333 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.253329 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.253336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:54.253688 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:54.254392 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:54.254419 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:54.256710 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:54.260325 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:54.260335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:54.260354 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:54.260870 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:54.261306 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:54.261310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:54.261313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:54.261559 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:54.262152 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:54.262159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:54.262510 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 12:49:54.262754 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 12:49:54.263171 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":289,"threads_started":1}
I20260924 12:49:54.265874 18097 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45209
I20260924 12:49:54.266191 18097 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:54.267028 18097 heartbeater.cc:511] Master 127.2.81.126:45209 requested a full tablet report, sending...
I20260924 12:49:54.269428 17908 ts_manager.cc:194] Registered new tserver with Master: f9be7d7f497c4d118192c20177a5c455 (127.2.81.65:45069)
I20260924 12:49:54.270887 17908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:38611
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:49:54.310287 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45209
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:54.643426 18103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:54.643725 18103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:54.643851 18103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:54.651371 18103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:54.651496 18103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:54.651640 18103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:49:54.661772 18103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:54.663059 18103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:54.664361 18103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:54.676950 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:54.677006 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:54.677027 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:54.677731 18103 server_base.cc:1034] running on GCE node
I20260924 12:49:54.678287 18103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:54.679212 18103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:54.680400 18103 hybrid_clock.cc:648] HybridClock initialized: now 1790254194680324 us; error 71 us; skew 500 ppm
I20260924 12:49:54.680750 18103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:54 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:54.683890 18103 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:49:54.685369 18103 webserver.cc:460] Webserver started at http://127.2.81.66:45371/ using document root <none> and password file <none>
I20260924 12:49:54.686041 18103 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:54.686156 18103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:54.686444 18103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:54.689494 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "066e8d71a11a4065aff4c16a8c3e522a"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "44b177374b96b8a7e6d53d539bd7130c"
server_key_iv: "029cc210f97036d5f1a7d12f097cc9f9"
server_key_version: "encryptionkey@0"
I20260924 12:49:54.690229 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "066e8d71a11a4065aff4c16a8c3e522a"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "44b177374b96b8a7e6d53d539bd7130c"
server_key_iv: "029cc210f97036d5f1a7d12f097cc9f9"
server_key_version: "encryptionkey@0"
I20260924 12:49:54.695492 18103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 12:49:54.699337 18118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:54.700822 18103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:49:54.701041 18103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "066e8d71a11a4065aff4c16a8c3e522a"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "44b177374b96b8a7e6d53d539bd7130c"
server_key_iv: "029cc210f97036d5f1a7d12f097cc9f9"
server_key_version: "encryptionkey@0"
I20260924 12:49:54.701247 18103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:54.718108 18103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:54.722442 18103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:54.722754 18103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:54.723721 18103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:54.725077 18103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:54.725152 18103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:54.725244 18103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:54.725286 18103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:54.756421 18103 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:41315
I20260924 12:49:54.756464 18230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:41315 every 8 connection(s)
I20260924 12:49:54.757929 18103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:49:54.763007 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18103
I20260924 12:49:54.763195 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:49:54.763717 2373 external_mini_cluster.cc:1428] Setting key 6e9b5d1d61bc928dccff1779b1fd3926
Sep 24 12:49:54 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:54.779752 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:54.760432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:57333 (local address 127.2.81.126:45209)
0924 12:49:54.760656 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:54.760661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:54.761894 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 12:49:54.764294 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:54.764299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:54.764422 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:54.764517 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:54.766586 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.767304 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.769045 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.769488 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.769606 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:54.773579 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:54.773586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:54.773589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:54.773634 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:54.776162 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:54.777192 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:54.777197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:54.777199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:54.777356 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:54.777895 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:54.777899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:54.777901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:54.778210 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:54.778335 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:54.779223 (+ 888us) server_negotiation.cc:299] Negotiation successful
0924 12:49:54.779420 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 12:49:54.781096 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:54.760922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:57333)
0924 12:49:54.761725 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:54.761770 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:54.763893 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:54.764998 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:54.765014 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:54.765786 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:54.766403 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:54.766424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.767482 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.767486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:54.768865 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:54.768883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:54.769733 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:54.769740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:54.769979 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:54.770647 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:54.770673 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:54.773212 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:54.776349 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:54.776360 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:54.776380 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:54.777044 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:54.777470 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:54.777475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:54.777478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:54.777742 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:54.778334 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:54.778342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:54.778791 (+ 449us) client_negotiation.cc:769] Sending connection context
0924 12:49:54.779090 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 12:49:54.779627 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":295,"threads_started":1}
I20260924 12:49:54.782702 18231 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45209
I20260924 12:49:54.783033 18231 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:54.783758 18231 heartbeater.cc:511] Master 127.2.81.126:45209 requested a full tablet report, sending...
I20260924 12:49:54.785266 17908 ts_manager.cc:194] Registered new tserver with Master: 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315)
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:49:54.786132 17908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:57333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:49:54.817353 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45209
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:49:55.154486 18236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:49:55.154804 18236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:49:55.154958 18236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:49:55.163522 18236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:49:55.163694 18236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:49:55.163851 18236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:49:55.174278 18236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:49:55.175598 18236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:49:55.176903 18236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:49:55.190778 18241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:55.190742 18242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:49:55.191236 18244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:49:55.191493 18236 server_base.cc:1034] running on GCE node
I20260924 12:49:55.192199 18236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:49:55.193300 18236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:49:55.194541 18236 hybrid_clock.cc:648] HybridClock initialized: now 1790254195194498 us; error 40 us; skew 500 ppm
I20260924 12:49:55.194921 18236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:49:55 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254195, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:49:55.198491 18236 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:49:55.200271 18236 webserver.cc:460] Webserver started at http://127.2.81.67:32807/ using document root <none> and password file <none>
I20260924 12:49:55.201092 18236 fs_manager.cc:359] Metadata directory not provided
I20260924 12:49:55.201231 18236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:49:55.201570 18236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:49:55.205047 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "20311c138ae745f88ff433a442ba40c3"
format_stamp: "Formatted at 2026-09-24 12:49:55 on dist-test-slave-9tm1"
server_key: "e3517a2e0152bfb7af6b4d2cad4a0494"
server_key_iv: "6d51c8c9b8fca1f3b2cac2b15736afba"
server_key_version: "encryptionkey@0"
I20260924 12:49:55.206202 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "20311c138ae745f88ff433a442ba40c3"
format_stamp: "Formatted at 2026-09-24 12:49:55 on dist-test-slave-9tm1"
server_key: "e3517a2e0152bfb7af6b4d2cad4a0494"
server_key_iv: "6d51c8c9b8fca1f3b2cac2b15736afba"
server_key_version: "encryptionkey@0"
I20260924 12:49:55.212118 18236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:49:55.216066 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:55.217698 18236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:49:55.217872 18236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "20311c138ae745f88ff433a442ba40c3"
format_stamp: "Formatted at 2026-09-24 12:49:55 on dist-test-slave-9tm1"
server_key: "e3517a2e0152bfb7af6b4d2cad4a0494"
server_key_iv: "6d51c8c9b8fca1f3b2cac2b15736afba"
server_key_version: "encryptionkey@0"
I20260924 12:49:55.218068 18236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:49:55.274724 18097 heartbeater.cc:503] Master 127.2.81.126:45209 was elected leader, sending a full tablet report...
I20260924 12:49:55.274852 18236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:49:55.280359 18236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:49:55.280699 18236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:49:55.282115 18236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:49:55.285274 18236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:49:55.285393 18236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:55.285569 18236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:49:55.285633 18236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:49:55.319532 18236 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:37243
I20260924 12:49:55.319764 18363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:37243 every 8 connection(s)
I20260924 12:49:55.320878 18236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:49:55.321738 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18236
I20260924 12:49:55.321938 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:49:55.322516 2373 external_mini_cluster.cc:1428] Setting key c97b50042b78959d8541670687602ebe
Sep 24 12:49:55 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254195, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:49:55.343225 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.323981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:57933 (local address 127.2.81.126:45209)
0924 12:49:55.324553 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.324557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.326052 (+ 1495us) server_negotiation.cc:407] Connection header received
0924 12:49:55.327720 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.327725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.327821 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.327915 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:55.329625 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.330401 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.332043 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.332473 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.332588 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.336584 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:55.336592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:55.336596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:55.336646 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:55.339879 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.340733 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.340737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.340739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.340868 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.341351 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.341354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.341356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.341646 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:55.341799 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.342533 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.342778 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
I20260924 12:49:55.344244 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.324669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:57933)
0924 12:49:55.325580 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:55.325637 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:55.327429 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:55.328181 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:55.328191 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:55.328741 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:49:55.329443 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.329465 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.330570 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.330574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.331852 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.331867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.332704 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.332711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.332958 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.333688 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:49:55.333714 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:49:55.336180 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:49:55.340066 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:55.340075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:55.340091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:55.340594 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:55.341015 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:49:55.341020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:49:55.341024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:49:55.341231 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:49:55.341777 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:49:55.341785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:49:55.342154 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 12:49:55.342409 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 12:49:55.342993 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":366,"threads_started":1}
I20260924 12:49:55.345873 18364 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45209
I20260924 12:49:55.346192 18364 heartbeater.cc:464] Registering TS with master...
I20260924 12:49:55.346940 18364 heartbeater.cc:511] Master 127.2.81.126:45209 requested a full tablet report, sending...
I20260924 12:49:55.348466 17908 ts_manager.cc:194] Registered new tserver with Master: 20311c138ae745f88ff433a442ba40c3 (127.2.81.67:37243)
I20260924 12:49:55.349393 17908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:57933
I20260924 12:49:55.352355 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:49:55.372277 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.357199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.2.81.126:45209)
0924 12:49:55.357375 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.357380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.357778 (+ 398us) server_negotiation.cc:407] Connection header received
0924 12:49:55.357894 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.357899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.357970 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.358067 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:55.359613 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.360434 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.361958 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.362427 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.362546 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.365311 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:55.365317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:55.365320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:55.365423 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:55.368816 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.369973 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.369980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.369984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.370133 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.370604 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.370610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.370614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.371102 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:55.371368 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.371704 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.371957 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 12:49:55.377763 17908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:49:55.385960 17908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:49:55.428510 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.400722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.2.81.66:41315)
0924 12:49:55.417602 (+ 16880us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.417613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.417650 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:49:55.417779 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.417788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.418125 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.420069 (+ 1944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:55.420445 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.421923 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.425665 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.427200 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.427330 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.427558 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.427989 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.428184 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16650,"thread_start_us":18148,"threads_started":1}
I20260924 12:49:55.428620 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.400165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:41315 (local address 127.0.0.1:53586)
0924 12:49:55.411693 (+ 11528us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:55.411764 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:55.412067 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:55.419270 (+ 7203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:55.419280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:55.419331 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:55.419807 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.419822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.422727 (+ 2905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.422732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.425448 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.425464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.427445 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.427453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.427698 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.427779 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 12:49:55.427928 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 12:49:55.428144 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11157,"mutex_wait_us":1,"thread_start_us":10948,"threads_started":1}
I20260924 12:49:55.430546 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.400736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.2.81.65:45069)
0924 12:49:55.418849 (+ 18113us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.418858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.421286 (+ 2428us) server_negotiation.cc:407] Connection header received
0924 12:49:55.421822 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.421831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.422142 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.422338 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:55.423353 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.424616 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.427196 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.429121 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.429252 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.429487 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.430165 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.430305 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17935,"thread_start_us":17801,"threads_started":1}
I20260924 12:49:55.431185 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.400533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:45069 (local address 127.0.0.1:51792)
0924 12:49:55.421144 (+ 20611us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:55.421167 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:55.421625 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:55.422618 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:55.422623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:55.422645 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:55.423126 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.423138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.424770 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.424780 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.426995 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.427012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.429651 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.429658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.429834 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.429867 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:49:55.429947 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 12:49:55.430904 (+ 957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20464,"mutex_wait_us":1,"thread_start_us":21635,"threads_started":1}
I20260924 12:49:55.432371 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.400165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:37243 (local address 127.0.0.1:34304)
0924 12:49:55.422438 (+ 22273us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:55.422459 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:55.422580 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:55.424464 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:55.424469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:55.424501 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:55.425129 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.425144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.427201 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.427206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.429623 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.429643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.431562 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.431570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.431738 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.431772 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:49:55.431865 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 12:49:55.432002 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22132,"mutex_wait_us":692,"thread_start_us":20443,"threads_started":1}
I20260924 12:49:55.432643 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.400571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.2.81.67:37243)
0924 12:49:55.423791 (+ 23220us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.423801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.423836 (+ 35us) server_negotiation.cc:407] Connection header received
0924 12:49:55.423947 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.423956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.424232 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.424451 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:55.425360 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.427026 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.429841 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.431267 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.431400 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.431631 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.431987 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.432164 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23041,"thread_start_us":22871,"threads_started":1}
I20260924 12:49:55.433041 18166 tablet_service.cc:1467] Processing CreateTablet for tablet 310b594c10f347bf845c45cca7e8a6be (DEFAULT_TABLE table=test-table [id=d852a3b2cc75454f9078730810afaef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:55.435405 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310b594c10f347bf845c45cca7e8a6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:55.435196 18032 tablet_service.cc:1467] Processing CreateTablet for tablet 310b594c10f347bf845c45cca7e8a6be (DEFAULT_TABLE table=test-table [id=d852a3b2cc75454f9078730810afaef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:55.436431 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310b594c10f347bf845c45cca7e8a6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:55.436482 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 310b594c10f347bf845c45cca7e8a6be (DEFAULT_TABLE table=test-table [id=d852a3b2cc75454f9078730810afaef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:49:55.437966 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310b594c10f347bf845c45cca7e8a6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:49:55.448333 18383 tablet_bootstrap.cc:492] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Bootstrap starting.
I20260924 12:49:55.450614 18384 tablet_bootstrap.cc:492] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Bootstrap starting.
I20260924 12:49:55.452617 18385 tablet_bootstrap.cc:492] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Bootstrap starting.
I20260924 12:49:55.455222 18383 tablet_bootstrap.cc:654] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:55.455259 18384 tablet_bootstrap.cc:654] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:55.456123 18385 tablet_bootstrap.cc:654] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Neither blocks nor log segments found. Creating new log.
I20260924 12:49:55.456599 18384 log.cc:824] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:55.456611 18383 log.cc:824] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:55.457186 18385 log.cc:824] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Log is configured to *not* fsync() on all Append() calls
I20260924 12:49:55.460059 18383 tablet_bootstrap.cc:492] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: No bootstrap required, opened a new log
I20260924 12:49:55.460362 18383 ts_tablet_manager.cc:1397] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 12:49:55.460471 18384 tablet_bootstrap.cc:492] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: No bootstrap required, opened a new log
I20260924 12:49:55.460721 18384 ts_tablet_manager.cc:1397] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 12:49:55.460822 18385 tablet_bootstrap.cc:492] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: No bootstrap required, opened a new log
I20260924 12:49:55.461118 18385 ts_tablet_manager.cc:1397] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 12:49:55.469373 18385 raft_consensus.cc:348] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.469792 18385 raft_consensus.cc:374] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:55.469882 18385 raft_consensus.cc:729] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20311c138ae745f88ff433a442ba40c3, State: Initialized, Role: FOLLOWER
I20260924 12:49:55.470486 18385 consensus_queue.cc:260] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.471081 18384 raft_consensus.cc:348] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.471566 18384 raft_consensus.cc:374] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:55.471609 18364 heartbeater.cc:503] Master 127.2.81.126:45209 was elected leader, sending a full tablet report...
I20260924 12:49:55.471377 18383 raft_consensus.cc:348] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.471657 18384 raft_consensus.cc:729] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066e8d71a11a4065aff4c16a8c3e522a, State: Initialized, Role: FOLLOWER
I20260924 12:49:55.471777 18383 raft_consensus.cc:374] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:49:55.471864 18383 raft_consensus.cc:729] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9be7d7f497c4d118192c20177a5c455, State: Initialized, Role: FOLLOWER
I20260924 12:49:55.472304 18385 ts_tablet_manager.cc:1428] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 12:49:55.472409 18383 consensus_queue.cc:260] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.472384 18384 consensus_queue.cc:260] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.473870 18231 heartbeater.cc:503] Master 127.2.81.126:45209 was elected leader, sending a full tablet report...
I20260924 12:49:55.474521 18384 ts_tablet_manager.cc:1428] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 12:49:55.476320 18383 ts_tablet_manager.cc:1428] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Time spent starting tablet: real 0.016s user 0.004s sys 0.010s
W20260924 12:49:55.495091 18098 tablet.cc:2369] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:49:55.509955 18232 tablet.cc:2369] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:55.545490 18391 raft_consensus.cc:482] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:49:55.545817 18391 raft_consensus.cc:504] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.547808 18391 leader_election.cc:288] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315), 20311c138ae745f88ff433a442ba40c3 (127.2.81.67:37243)
I20260924 12:49:55.555168 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.548526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:32941 (local address 127.2.81.67:37243)
0924 12:49:55.548724 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.548730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.549382 (+ 652us) server_negotiation.cc:407] Connection header received
0924 12:49:55.549681 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.549688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.549763 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.549850 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:55.550421 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.551266 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.553291 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.554107 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.554209 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.554294 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.554754 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.554902 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:49:55.555404 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.548181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:41315 (local address 127.2.81.65:51051)
0924 12:49:55.549590 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:55.549608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:55.549739 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:55.550064 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:55.550067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:55.550090 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:55.550457 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.550469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.551466 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.551471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.553165 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.553179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.554541 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.554546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.554743 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.554777 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:49:55.554878 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 12:49:55.555002 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"mutex_wait_us":4,"thread_start_us":706,"threads_started":1}
I20260924 12:49:55.555511 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.548109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:37243 (local address 127.2.81.65:32941)
0924 12:49:55.549274 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0924 12:49:55.549294 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:49:55.549508 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:49:55.549877 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:49:55.549882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:49:55.549906 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:49:55.550273 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.550284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.551399 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.551405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.553119 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:49:55.553135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.554347 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.554353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:49:55.554551 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.554587 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 12:49:55.554688 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 12:49:55.554845 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":5,"thread_start_us":1214,"threads_started":1}
I20260924 12:49:55.555553 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.548664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:51051 (local address 127.2.81.66:41315)
0924 12:49:55.548894 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.548900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.549706 (+ 806us) server_negotiation.cc:407] Connection header received
0924 12:49:55.549858 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.549863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.549939 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.550035 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:49:55.550626 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.551316 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.553347 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.554296 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.554450 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.554544 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.554910 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.555045 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 12:49:55.556437 18186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310b594c10f347bf845c45cca7e8a6be" candidate_uuid: "f9be7d7f497c4d118192c20177a5c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066e8d71a11a4065aff4c16a8c3e522a" is_pre_election: true
I20260924 12:49:55.556581 18319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310b594c10f347bf845c45cca7e8a6be" candidate_uuid: "f9be7d7f497c4d118192c20177a5c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20311c138ae745f88ff433a442ba40c3" is_pre_election: true
I20260924 12:49:55.556949 18186 raft_consensus.cc:2454] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9be7d7f497c4d118192c20177a5c455 in term 0.
I20260924 12:49:55.557087 18319 raft_consensus.cc:2454] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9be7d7f497c4d118192c20177a5c455 in term 0.
I20260924 12:49:55.557703 17987 leader_election.cc:302] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066e8d71a11a4065aff4c16a8c3e522a, f9be7d7f497c4d118192c20177a5c455; no voters:
I20260924 12:49:55.558193 18391 raft_consensus.cc:2781] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:49:55.558343 18391 raft_consensus.cc:482] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:49:55.558454 18391 raft_consensus.cc:3037] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:55.560809 18391 raft_consensus.cc:504] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.561473 18391 leader_election.cc:288] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [CANDIDATE]: Term 1 election: Requested vote from peers 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315), 20311c138ae745f88ff433a442ba40c3 (127.2.81.67:37243)
I20260924 12:49:55.562096 18319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310b594c10f347bf845c45cca7e8a6be" candidate_uuid: "f9be7d7f497c4d118192c20177a5c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20311c138ae745f88ff433a442ba40c3"
I20260924 12:49:55.562096 18186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310b594c10f347bf845c45cca7e8a6be" candidate_uuid: "f9be7d7f497c4d118192c20177a5c455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066e8d71a11a4065aff4c16a8c3e522a"
I20260924 12:49:55.562325 18186 raft_consensus.cc:3037] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:55.562325 18319 raft_consensus.cc:3037] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:49:55.565544 18186 raft_consensus.cc:2454] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9be7d7f497c4d118192c20177a5c455 in term 1.
I20260924 12:49:55.565544 18319 raft_consensus.cc:2454] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9be7d7f497c4d118192c20177a5c455 in term 1.
I20260924 12:49:55.566082 17987 leader_election.cc:302] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 066e8d71a11a4065aff4c16a8c3e522a, 20311c138ae745f88ff433a442ba40c3, f9be7d7f497c4d118192c20177a5c455; no voters:
I20260924 12:49:55.566447 18391 raft_consensus.cc:2781] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:49:55.566963 18391 raft_consensus.cc:686] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 1 LEADER]: Becoming Leader. State: Replica: f9be7d7f497c4d118192c20177a5c455, State: Running, Role: LEADER
I20260924 12:49:55.567444 18391 consensus_queue.cc:237] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } }
I20260924 12:49:55.572099 17906 catalog_manager.cc:5495] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9be7d7f497c4d118192c20177a5c455 (127.2.81.65). New cstate: current_term: 1 leader_uuid: "f9be7d7f497c4d118192c20177a5c455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } health_report { overall_health: HEALTHY } } }
W20260924 12:49:55.572619 18365 tablet.cc:2369] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:49:55.592559 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.586547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.2.81.65:45069)
0924 12:49:55.586731 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.586739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.586759 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:49:55.586935 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.586938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.587035 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.587169 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:49:55.587608 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.588470 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.590245 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.590725 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.590835 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.591307 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:49:55.591890 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:49:55.591999 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.592156 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.592235 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 12:49:55.601538 18319 raft_consensus.cc:1261] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Refusing update from remote peer f9be7d7f497c4d118192c20177a5c455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:55.601538 18186 raft_consensus.cc:1261] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Refusing update from remote peer f9be7d7f497c4d118192c20177a5c455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:49:55.602783 18394 consensus_queue.cc:1035] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:55.603178 18391 consensus_queue.cc:1035] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:49:55.618110 18398 mvcc.cc:205] Tried to move back new op lower bound from 7332881185171673088 to 7332881185047994368. Current Snapshot: MvccSnapshot[applied={T|T < 7332881185171673088}]
I20260924 12:49:55.641991 17907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:55.642263 17907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:49:55.646119 17907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Sending DeleteTablet for 3 replicas of tablet 310b594c10f347bf845c45cca7e8a6be
I20260924 12:49:55.647352 18166 tablet_service.cc:1514] Processing DeleteTablet for tablet 310b594c10f347bf845c45cca7e8a6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:55 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:53586
I20260924 12:49:55.647356 18299 tablet_service.cc:1514] Processing DeleteTablet for tablet 310b594c10f347bf845c45cca7e8a6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:55 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:34304
I20260924 12:49:55.647639 18032 tablet_service.cc:1514] Processing DeleteTablet for tablet 310b594c10f347bf845c45cca7e8a6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:49:55 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:51792
I20260924 12:49:55.648736 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17969
I20260924 12:49:55.661600 18408 tablet_replica.cc:330] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: stopping tablet replica
I20260924 12:49:55.662182 18408 raft_consensus.cc:2229] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:49:55.662751 18406 tablet_replica.cc:330] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: stopping tablet replica
I20260924 12:49:55.663043 18408 raft_consensus.cc:2258] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:55.663600 18406 raft_consensus.cc:2229] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:55.665210 18406 raft_consensus.cc:2258] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:55.665664 18408 ts_tablet_manager.cc:1905] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:55.670882 18406 ts_tablet_manager.cc:1905] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:55.671556 18407 tablet_replica.cc:330] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: stopping tablet replica
I20260924 12:49:55.672009 18407 raft_consensus.cc:2229] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:49:55.672546 18407 raft_consensus.cc:2258] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:49:55.673623 18408 ts_tablet_manager.cc:1918] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:55.673818 18408 log.cc:1196] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/310b594c10f347bf845c45cca7e8a6be
I20260924 12:49:55.675096 18407 ts_tablet_manager.cc:1905] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:49:55.675776 18408 ts_tablet_manager.cc:1939] T 310b594c10f347bf845c45cca7e8a6be P f9be7d7f497c4d118192c20177a5c455: Deleting consensus metadata
Sep 24 12:49:55 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:49:55.682287 17893 catalog_manager.cc:4841] TS f9be7d7f497c4d118192c20177a5c455 (127.2.81.65:45069): tablet 310b594c10f347bf845c45cca7e8a6be (table test-table [id=d852a3b2cc75454f9078730810afaef0]) successfully deleted
I20260924 12:49:55.682351 18406 ts_tablet_manager.cc:1918] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:55.682538 18406 log.cc:1196] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/310b594c10f347bf845c45cca7e8a6be
I20260924 12:49:55.683099 18406 ts_tablet_manager.cc:1939] T 310b594c10f347bf845c45cca7e8a6be P 066e8d71a11a4065aff4c16a8c3e522a: Deleting consensus metadata
I20260924 12:49:55.683315 18407 ts_tablet_manager.cc:1918] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:49:55.683557 18407 log.cc:1196] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/310b594c10f347bf845c45cca7e8a6be
I20260924 12:49:55.684142 18407 ts_tablet_manager.cc:1939] T 310b594c10f347bf845c45cca7e8a6be P 20311c138ae745f88ff433a442ba40c3: Deleting consensus metadata
I20260924 12:49:55.685295 17894 catalog_manager.cc:4841] TS 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315): tablet 310b594c10f347bf845c45cca7e8a6be (table test-table [id=d852a3b2cc75454f9078730810afaef0]) successfully deleted
I20260924 12:49:55.686239 17895 catalog_manager.cc:4841] TS 20311c138ae745f88ff433a442ba40c3 (127.2.81.67:37243): tablet 310b594c10f347bf845c45cca7e8a6be (table test-table [id=d852a3b2cc75454f9078730810afaef0]) successfully deleted
I20260924 12:49:55.691495 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.650169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.2.81.65:45069)
0924 12:49:55.650363 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.650369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.658605 (+ 8236us) server_negotiation.cc:407] Connection header received
0924 12:49:55.659958 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.659966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.660079 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.660162 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:55.665463 (+ 5301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.666571 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.673450 (+ 6879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.674369 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.674531 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.684318 (+ 9787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:55.684338 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:55.684359 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:55.684441 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:55.688271 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.689224 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.689232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.689235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.689399 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.689837 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.689840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.689843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.690234 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:55.690532 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.690911 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.691172 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:49:55.692212 18092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:55.792896 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17969
I20260924 12:49:55.808501 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 18103
Sep 24 12:49:55 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:49:55.824558 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.809365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.2.81.66:41315)
0924 12:49:55.809564 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.809569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.809606 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:49:55.809661 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.809665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.809745 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.809835 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:55.811155 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.811957 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.813584 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.814025 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.814141 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.818614 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:55.818637 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:55.818653 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:55.818695 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:55.821617 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.822434 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.822442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.822444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.822593 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.823052 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.823056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.823058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.823545 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:55.823794 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.824104 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.824289 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 12:49:55.825253 18226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:55.921006 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 18103
I20260924 12:49:55.935039 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 18236
Sep 24 12:49:55 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:49:55.951165 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:49:55.935785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.2.81.67:37243)
0924 12:49:55.935940 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 12:49:55.935944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:49:55.935963 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:49:55.936121 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:49:55.936126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:49:55.936207 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:49:55.936304 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:49:55.937819 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.938850 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.940192 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:49:55.940616 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:49:55.940740 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:49:55.945295 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:49:55.945313 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:49:55.945329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:49:55.945368 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:49:55.948211 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.949130 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.949137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.949139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.949286 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:49:55.949692 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:49:55.949695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:49:55.949697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:49:55.950035 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:49:55.950280 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 12:49:55.950615 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 12:49:55.950803 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 12:49:55.951747 18359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:56.042964 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 18236
I20260924 12:49:56.056209 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 17876
I20260924 12:49:56.057818 17938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:49:56.170770 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 17876
I20260924 12:50:16.189029 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45209
--webserver_interface=127.2.81.126
--webserver_port=39341
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:50:16.516558 18414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:16.516902 18414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:16.517052 18414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:16.525113 18414 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 12:50:16.525233 18414 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 12:50:16.525311 18414 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 12:50:16.525390 18414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:50:16.525473 18414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:16.525568 18414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:16.525633 18414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:50:16.525707 18414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:50:16.536074 18414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:16.537423 18414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:16.538756 18414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:16.562254 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:16.562297 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:16.562297 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:16.563092 18414 server_base.cc:1034] running on GCE node
I20260924 12:50:16.563992 18414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:16.565699 18414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:16.566932 18414 hybrid_clock.cc:648] HybridClock initialized: now 1790254216566874 us; error 29 us; skew 500 ppm
I20260924 12:50:16.567327 18414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:16 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:16.571000 18414 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:50:16.572674 18414 webserver.cc:460] Webserver started at http://127.2.81.126:39341/ using document root <none> and password file <none>
I20260924 12:50:16.573485 18414 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:16.573611 18414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:16.579563 18414 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:50:16.582566 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:16.584020 18414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:50:16.584298 18414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "b999ae8319154608b62dffa05c9d82f9"
format_stamp: "Formatted at 2026-09-24 12:49:53 on dist-test-slave-9tm1"
server_key: "12ac9b933c451b2bbdb09efb59342997"
server_key_iv: "e817d4a5d20483e99ed774388697e723"
server_key_version: "encryptionkey@0"
I20260924 12:50:16.585464 18414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:16.614856 18414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:16.619570 18414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:16.619858 18414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:16.636656 18414 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45209
I20260924 12:50:16.636685 18481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45209 every 8 connection(s)
I20260924 12:50:16.638126 18414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:50:16.642547 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18414
I20260924 12:50:16.643323 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:45069
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:45209
--webserver_port=36031
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:50:16.645326 18482 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:50:16.648909 18482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap starting.
I20260924 12:50:16.668870 18482 log.cc:824] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:50:16.681682 18482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:50:16.682224 18482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap complete.
I20260924 12:50:16.691690 18482 raft_consensus.cc:348] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:16.692590 18482 raft_consensus.cc:729] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b999ae8319154608b62dffa05c9d82f9, State: Initialized, Role: FOLLOWER
I20260924 12:50:16.693218 18482 consensus_queue.cc:260] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:16.693462 18482 raft_consensus.cc:388] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:50:16.693576 18482 raft_consensus.cc:482] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:50:16.693728 18482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 12:50:16.697651 18482 raft_consensus.cc:504] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:16.698256 18482 leader_election.cc:302] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b999ae8319154608b62dffa05c9d82f9; no voters:
I20260924 12:50:16.698933 18482 leader_election.cc:288] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 12:50:16.699054 18486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 12:50:16.700208 18486 raft_consensus.cc:686] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 LEADER]: Becoming Leader. State: Replica: b999ae8319154608b62dffa05c9d82f9, State: Running, Role: LEADER
I20260924 12:50:16.700954 18486 consensus_queue.cc:237] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:16.701249 18482 sys_catalog.cc:564] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:50:16.703442 18488 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b999ae8319154608b62dffa05c9d82f9. Latest consensus state: current_term: 2 leader_uuid: "b999ae8319154608b62dffa05c9d82f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } } }
I20260924 12:50:16.703819 18488 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:50:16.704382 18487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b999ae8319154608b62dffa05c9d82f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } } }
I20260924 12:50:16.704692 18487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:50:16.709465 18493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:50:16.718233 18493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:50:16.719827 18493 catalog_manager.cc:1259] Loaded cluster ID: 5f2252a736e94ae084748f63738e0e50
I20260924 12:50:16.719995 18493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:50:16.724004 18493 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:50:16.725715 18493 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Loaded TSK: 0
I20260924 12:50:16.758958 18493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Generated new TSK 1
I20260924 12:50:16.759431 18493 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 12:50:16.986985 18484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:16.987293 18484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:16.987437 18484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:16.995126 18484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:16.995261 18484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:16.995424 18484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:50:17.006282 18484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:45069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=36031
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:17.007763 18484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:17.009258 18484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:17.024293 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:17.024273 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:17.024865 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:17.026273 18484 server_base.cc:1034] running on GCE node
I20260924 12:50:17.027150 18484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:17.028532 18484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:17.029795 18484 hybrid_clock.cc:648] HybridClock initialized: now 1790254217029740 us; error 44 us; skew 500 ppm
I20260924 12:50:17.030210 18484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:17 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:17.034298 18484 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:50:17.035907 18484 webserver.cc:460] Webserver started at http://127.2.81.65:36031/ using document root <none> and password file <none>
I20260924 12:50:17.036572 18484 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:17.036695 18484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:17.042555 18484 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:50:17.045554 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.046972 18484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:50:17.047149 18484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "f9be7d7f497c4d118192c20177a5c455"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "295fb04499c834161d0a6a56f37f9db2"
server_key_iv: "659b38f6d14e11b0b38120421e3b3858"
server_key_version: "encryptionkey@0"
I20260924 12:50:17.048207 18484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:17.062687 18484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:17.067195 18484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:17.067473 18484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:17.068521 18484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:17.069960 18484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:17.070060 18484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.070178 18484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:17.070250 18484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.104173 18484 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:45069
I20260924 12:50:17.104184 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:45069 every 8 connection(s)
I20260924 12:50:17.106045 18484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:50:17.116737 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18484
I20260924 12:50:17.117624 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:41315
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:45209
--webserver_port=45371
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:50:17 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:17.133759 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:17.109428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:46247)
0924 12:50:17.110305 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:17.110342 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:17.112114 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:17.113755 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:17.113772 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:17.114329 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:50:17.114875 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:17.114892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.116857 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.116863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:17.118917 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:17.118936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.120262 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.120270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:17.120540 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:17.121388 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:50:17.121419 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:50:17.124612 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:50:17.129151 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:17.129161 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:17.129175 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:17.129706 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:17.130298 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:17.130304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:17.130308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:17.130605 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:17.131608 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:50:17.131615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:50:17.131943 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 12:50:17.132205 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 12:50:17.132645 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260924 12:50:17.134065 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:17.109301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:46247 (local address 127.2.81.126:45209)
0924 12:50:17.110028 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:17.110045 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:17.110483 (+ 438us) server_negotiation.cc:407] Connection header received
0924 12:50:17.112717 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:17.112732 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:17.113197 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:17.113541 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:17.115093 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.116637 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.119122 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.120006 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.120177 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:17.125050 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:17.125063 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:17.125084 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:17.125139 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:17.128876 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:17.129896 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:17.129906 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:17.129913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:17.130144 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:17.130731 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:17.130736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:17.130739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:17.131450 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:17.131648 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:17.132471 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 12:50:17.132941 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":303,"threads_started":1}
I20260924 12:50:17.137429 18632 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45209
I20260924 12:50:17.137795 18632 heartbeater.cc:464] Registering TS with master...
I20260924 12:50:17.138595 18632 heartbeater.cc:511] Master 127.2.81.126:45209 requested a full tablet report, sending...
I20260924 12:50:17.141736 18447 ts_manager.cc:194] Registered new tserver with Master: f9be7d7f497c4d118192c20177a5c455 (127.2.81.65:45069)
I20260924 12:50:17.143365 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:46247
W20260924 12:50:17.456601 18636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:17.456897 18636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:17.457067 18636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:17.464466 18636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:17.464597 18636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:17.464757 18636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:50:17.475099 18636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:41315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=45371
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:17.476505 18636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:17.477888 18636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:17.491716 18642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:17.492134 18636 server_base.cc:1034] running on GCE node
W20260924 12:50:17.491755 18644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:17.491783 18641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:17.493119 18636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:17.494123 18636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:17.495352 18636 hybrid_clock.cc:648] HybridClock initialized: now 1790254217495283 us; error 45 us; skew 500 ppm
I20260924 12:50:17.495711 18636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:17 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:17.499845 18636 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:50:17.501577 18636 webserver.cc:460] Webserver started at http://127.2.81.66:45371/ using document root <none> and password file <none>
I20260924 12:50:17.502331 18636 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:17.502480 18636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:17.508311 18636 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 12:50:17.511416 18651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.512795 18636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:50:17.512938 18636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "066e8d71a11a4065aff4c16a8c3e522a"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "44b177374b96b8a7e6d53d539bd7130c"
server_key_iv: "029cc210f97036d5f1a7d12f097cc9f9"
server_key_version: "encryptionkey@0"
I20260924 12:50:17.513989 18636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:17.528754 18636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:17.533043 18636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:17.533298 18636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:17.534315 18636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:17.535645 18636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:17.535732 18636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.535815 18636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:17.535866 18636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.570919 18636 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:41315
I20260924 12:50:17.570994 18763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:41315 every 8 connection(s)
I20260924 12:50:17.572587 18636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:50:17.576824 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18636
I20260924 12:50:17.577589 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:37243
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:45209
--webserver_port=32807
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 12:50:17 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:17.595989 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:17.575304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44187 (local address 127.2.81.126:45209)
0924 12:50:17.575485 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:17.575490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:17.576846 (+ 1356us) server_negotiation.cc:407] Connection header received
0924 12:50:17.579500 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:17.579505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:17.579582 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:17.579674 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:17.581745 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.582457 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.584641 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.585182 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.585305 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:17.589483 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:17.589490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:17.589494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:17.589539 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:17.592652 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:17.593598 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:17.593603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:17.593606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:17.593748 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:17.594176 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:17.594181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:17.594183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:17.594516 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:17.594655 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:17.595398 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 12:50:17.595648 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:50:17.596911 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:17.575759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:44187)
0924 12:50:17.576663 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:17.576722 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:17.579116 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:17.579948 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:17.579960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:17.580689 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:50:17.581524 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:17.581550 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.582624 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.582629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:17.584455 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:17.584471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:17.585425 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:17.585433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:17.585735 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:17.586541 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:50:17.586566 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:50:17.589146 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:50:17.592839 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:17.592848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:17.592863 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:17.593429 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:17.593856 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:17.593859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:17.593862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:17.594061 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:17.594645 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:50:17.594652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:50:17.595031 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 12:50:17.595276 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 12:50:17.595694 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260924 12:50:17.598569 18764 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45209
I20260924 12:50:17.598869 18764 heartbeater.cc:464] Registering TS with master...
I20260924 12:50:17.599619 18764 heartbeater.cc:511] Master 127.2.81.126:45209 requested a full tablet report, sending...
I20260924 12:50:17.601259 18447 ts_manager.cc:194] Registered new tserver with Master: 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315)
I20260924 12:50:17.602427 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:44187
W20260924 12:50:17.912344 18767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:17.912647 18767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:17.912782 18767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:17.920217 18767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:17.920351 18767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:17.920529 18767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:50:17.930608 18767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:37243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=32807
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:17.931924 18767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:17.933275 18767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:17.946962 18773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:17.946969 18772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:17.946894 18775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:17.947654 18767 server_base.cc:1034] running on GCE node
I20260924 12:50:17.948333 18767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:17.949395 18767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:17.950598 18767 hybrid_clock.cc:648] HybridClock initialized: now 1790254217950548 us; error 56 us; skew 500 ppm
I20260924 12:50:17.950903 18767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:17 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:17.959444 18767 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:50:17.960954 18767 webserver.cc:460] Webserver started at http://127.2.81.67:32807/ using document root <none> and password file <none>
I20260924 12:50:17.961658 18767 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:17.961772 18767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:17.967278 18767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 12:50:17.970125 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:17.971575 18767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:50:17.971730 18767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "20311c138ae745f88ff433a442ba40c3"
format_stamp: "Formatted at 2026-09-24 12:49:55 on dist-test-slave-9tm1"
server_key: "e3517a2e0152bfb7af6b4d2cad4a0494"
server_key_iv: "6d51c8c9b8fca1f3b2cac2b15736afba"
server_key_version: "encryptionkey@0"
I20260924 12:50:17.972858 18767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:17.994333 18767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:17.998653 18767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:17.998916 18767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:17.999891 18767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:18.001304 18767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:18.001397 18767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:18.001483 18767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:18.001550 18767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:18.031785 18767 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:37243
I20260924 12:50:18.031865 18894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:37243 every 8 connection(s)
I20260924 12:50:18.033270 18767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:50:18.035003 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18767
Sep 24 12:50:18 dist-test-slave-9tm1 krb5kdc[17863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254217, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:18.051179 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.035938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.2.81.126:45209)
0924 12:50:18.036778 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.036785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.036807 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:50:18.036874 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.036879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.036958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.037105 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:18.038756 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.039872 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.041328 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.041847 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.041948 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.045030 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:18.045037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:18.045042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:18.045081 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:18.048002 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.048862 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.048867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.048870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.049097 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.049497 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.049502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.049506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.049864 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:18.050156 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.050501 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.050701 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":733,"thread_start_us":457,"threads_started":1}
I20260924 12:50:18.056143 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.035910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:37341 (local address 127.2.81.126:45209)
0924 12:50:18.036102 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.036108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.037424 (+ 1316us) server_negotiation.cc:407] Connection header received
0924 12:50:18.039580 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.039584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.039654 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.039748 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:18.041401 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.042286 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.043888 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.044525 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.044656 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.048859 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:18.048866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:18.048872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:18.048939 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:18.052198 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.053420 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.053427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.053429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.053628 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.054128 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.054134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.054137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.054465 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:18.054635 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.055578 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.055782 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":11136}
I20260924 12:50:18.057107 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.036420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:37341)
0924 12:50:18.037289 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:18.037331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:18.039212 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:18.040044 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:18.040053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:18.040592 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:50:18.041239 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.041263 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.042453 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.042457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.043732 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.043745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.044775 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.044781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.045122 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.045829 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:50:18.045865 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:50:18.048581 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:50:18.052434 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:18.052461 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:18.052482 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:18.053269 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:18.053747 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:18.053752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:18.053756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:18.054009 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:18.054661 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:50:18.054668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:50:18.055154 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 12:50:18.055423 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 12:50:18.055865 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":328,"threads_started":1}
I20260924 12:50:18.058640 18895 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45209
I20260924 12:50:18.058976 18895 heartbeater.cc:464] Registering TS with master...
I20260924 12:50:18.059725 18895 heartbeater.cc:511] Master 127.2.81.126:45209 requested a full tablet report, sending...
I20260924 12:50:18.061429 18447 ts_manager.cc:194] Registered new tserver with Master: 20311c138ae745f88ff433a442ba40c3 (127.2.81.67:37243)
I20260924 12:50:18.062255 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:37341
I20260924 12:50:18.066731 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:50:18.073601 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.067668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.2.81.126:45209)
0924 12:50:18.067882 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.067889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.068283 (+ 394us) server_negotiation.cc:407] Connection header received
0924 12:50:18.068518 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.068565 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.068714 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.068854 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:50:18.069361 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.070140 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.071637 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.072050 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.072162 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.072673 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:50:18.072886 (+ 213us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 12:50:18.073303 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40268: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 12:50:18.073730 18634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40268: token expired
W20260924 12:50:18.074324 18371 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.81.126:45209)
I20260924 12:50:18.088896 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.074859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.2.81.126:45209)
0924 12:50:18.075027 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.075031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.075048 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:50:18.075132 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.075135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.075196 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.075273 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:18.076940 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.077919 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.079547 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.079955 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.080063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.083234 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:18.083245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:18.083250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:18.083295 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:18.086222 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.087034 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.087040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.087044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.087174 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.087472 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.087475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.087477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.087856 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:18.088041 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.088343 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.088568 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 12:50:18.092505 18447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:50:18.099715 18447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:50:18.121703 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.112007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:41315 (local address 127.0.0.1:35830)
0924 12:50:18.112765 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:18.112805 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:18.113027 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:18.114199 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:18.114206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:18.114239 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:50:18.114686 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.114699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.116489 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.116493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.118852 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.118866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.120722 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.120729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.120961 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.121053 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 12:50:18.121179 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 12:50:18.121361 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":547,"threads_started":1}
I20260924 12:50:18.122355 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.112129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.2.81.66:41315)
0924 12:50:18.113567 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.113577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.113609 (+ 32us) server_negotiation.cc:407] Connection header received
0924 12:50:18.113712 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.113720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.113977 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.114182 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:50:18.114912 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.116334 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.119037 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.120495 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.120627 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.120815 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.121842 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.122024 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":1155,"threads_started":1}
I20260924 12:50:18.125797 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.112166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:45069 (local address 127.0.0.1:51322)
0924 12:50:18.115185 (+ 3019us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:18.115204 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:18.115400 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:18.116289 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:18.116295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:18.116321 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:50:18.116788 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.116799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.118965 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.118970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.121082 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.121094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.123235 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.123241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.125151 (+ 1910us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.125186 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:50:18.125280 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 12:50:18.125418 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2904,"mutex_wait_us":1672,"thread_start_us":3744,"threads_started":1}
I20260924 12:50:18.125869 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.112477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.2.81.65:45069)
0924 12:50:18.114787 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.114795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.115295 (+ 500us) server_negotiation.cc:407] Connection header received
0924 12:50:18.115607 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.115617 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.115841 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.116021 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:50:18.116995 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.118808 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.121227 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.122635 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.122781 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.123021 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.125369 (+ 2348us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.125562 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":5135,"threads_started":1}
I20260924 12:50:18.126314 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.114911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:37243 (local address 127.0.0.1:34876)
0924 12:50:18.118293 (+ 3382us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:18.118307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:18.118419 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:18.119375 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:18.119380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:18.119403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:50:18.119913 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.119944 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.121810 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.121817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.123866 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.123880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.125480 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.125487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.125650 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.125688 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:50:18.125780 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 12:50:18.125921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3261,"mutex_wait_us":50,"thread_start_us":3103,"threads_started":1}
I20260924 12:50:18.125996 18699 tablet_service.cc:1467] Processing CreateTablet for tablet dabd801bab4749e883be89ddeca475ca (DEFAULT_TABLE table=test-table [id=90812385b7ee4d6d81f7f31f105befe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:50:18.127102 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.115455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.2.81.67:37243)
0924 12:50:18.117699 (+ 2244us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.117707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.118413 (+ 706us) server_negotiation.cc:407] Connection header received
0924 12:50:18.118565 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.118570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.118903 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.119108 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:50:18.120123 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.121620 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.124035 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.124966 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.125111 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.125325 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.126329 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.126790 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2046,"thread_start_us":1904,"threads_started":1}
I20260924 12:50:18.127511 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabd801bab4749e883be89ddeca475ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:50:18.129511 18567 tablet_service.cc:1467] Processing CreateTablet for tablet dabd801bab4749e883be89ddeca475ca (DEFAULT_TABLE table=test-table [id=90812385b7ee4d6d81f7f31f105befe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:50:18.130661 18567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabd801bab4749e883be89ddeca475ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:50:18.130448 18830 tablet_service.cc:1467] Processing CreateTablet for tablet dabd801bab4749e883be89ddeca475ca (DEFAULT_TABLE table=test-table [id=90812385b7ee4d6d81f7f31f105befe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:50:18.131753 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabd801bab4749e883be89ddeca475ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:50:18.143543 18911 tablet_bootstrap.cc:492] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Bootstrap starting.
I20260924 12:50:18.146749 18632 heartbeater.cc:503] Master 127.2.81.126:45209 was elected leader, sending a full tablet report...
I20260924 12:50:18.146893 18913 tablet_bootstrap.cc:492] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Bootstrap starting.
I20260924 12:50:18.147106 18911 tablet_bootstrap.cc:654] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Neither blocks nor log segments found. Creating new log.
I20260924 12:50:18.148614 18911 log.cc:824] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Log is configured to *not* fsync() on all Append() calls
I20260924 12:50:18.150014 18912 tablet_bootstrap.cc:492] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Bootstrap starting.
I20260924 12:50:18.150906 18913 tablet_bootstrap.cc:654] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Neither blocks nor log segments found. Creating new log.
I20260924 12:50:18.152315 18913 log.cc:824] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Log is configured to *not* fsync() on all Append() calls
I20260924 12:50:18.154978 18912 tablet_bootstrap.cc:654] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Neither blocks nor log segments found. Creating new log.
I20260924 12:50:18.155910 18913 tablet_bootstrap.cc:492] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: No bootstrap required, opened a new log
I20260924 12:50:18.155925 18911 tablet_bootstrap.cc:492] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: No bootstrap required, opened a new log
I20260924 12:50:18.155997 18912 log.cc:824] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:50:18.156198 18913 ts_tablet_manager.cc:1397] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260924 12:50:18.156198 18911 ts_tablet_manager.cc:1397] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260924 12:50:18.158504 18912 tablet_bootstrap.cc:492] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: No bootstrap required, opened a new log
I20260924 12:50:18.158773 18912 ts_tablet_manager.cc:1397] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 12:50:18.166646 18912 raft_consensus.cc:348] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.167057 18912 raft_consensus.cc:374] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:50:18.167150 18912 raft_consensus.cc:729] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066e8d71a11a4065aff4c16a8c3e522a, State: Initialized, Role: FOLLOWER
I20260924 12:50:18.167143 18911 raft_consensus.cc:348] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.167143 18913 raft_consensus.cc:348] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.167570 18911 raft_consensus.cc:374] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:50:18.167570 18913 raft_consensus.cc:374] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:50:18.167680 18911 raft_consensus.cc:729] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9be7d7f497c4d118192c20177a5c455, State: Initialized, Role: FOLLOWER
I20260924 12:50:18.167675 18913 raft_consensus.cc:729] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20311c138ae745f88ff433a442ba40c3, State: Initialized, Role: FOLLOWER
I20260924 12:50:18.167776 18912 consensus_queue.cc:260] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.168251 18911 consensus_queue.cc:260] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.168273 18913 consensus_queue.cc:260] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.169021 18764 heartbeater.cc:503] Master 127.2.81.126:45209 was elected leader, sending a full tablet report...
I20260924 12:50:18.169670 18912 ts_tablet_manager.cc:1428] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260924 12:50:18.171715 18913 ts_tablet_manager.cc:1428] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20260924 12:50:18.172403 18895 heartbeater.cc:503] Master 127.2.81.126:45209 was elected leader, sending a full tablet report...
I20260924 12:50:18.172863 18911 ts_tablet_manager.cc:1428] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260924 12:50:18.256644 18918 raft_consensus.cc:482] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:50:18.256951 18918 raft_consensus.cc:504] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.258744 18918 leader_election.cc:288] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315), f9be7d7f497c4d118192c20177a5c455 (127.2.81.65:45069)
I20260924 12:50:18.264817 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.259193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:45069 (local address 127.2.81.67:60617)
0924 12:50:18.259386 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:18.259405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:18.259526 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:18.259872 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:18.259877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:18.259917 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:50:18.260130 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.260139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.261373 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.261376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.263066 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.263080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.264246 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.264250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.264380 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.264405 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:50:18.264478 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 12:50:18.264614 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 12:50:18.264917 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.259275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:60617 (local address 127.2.81.65:45069)
0924 12:50:18.259494 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.259499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.259518 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:50:18.259645 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.259649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.259722 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.259818 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:50:18.260300 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.261241 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.263244 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.264085 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.264221 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.264340 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.264541 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.264677 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 12:50:18.266105 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.259287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:47311 (local address 127.2.81.66:41315)
0924 12:50:18.259485 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.259492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.260425 (+ 933us) server_negotiation.cc:407] Connection header received
0924 12:50:18.260567 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.260573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.260673 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.260757 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:50:18.261326 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.262015 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.263972 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.265207 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.265356 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.265444 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.265735 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.265860 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:50:18.266180 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.259193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:41315 (local address 127.2.81.67:47311)
0924 12:50:18.260324 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:18.260340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:18.260452 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:18.260801 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:18.260806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:18.260831 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:50:18.261175 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.261183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.262163 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.262169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.263801 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:18.263813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.265417 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.265423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:18.265575 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.265600 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:50:18.265682 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 12:50:18.265804 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":403,"thread_start_us":406,"threads_started":1}
I20260924 12:50:18.265965 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabd801bab4749e883be89ddeca475ca" candidate_uuid: "20311c138ae745f88ff433a442ba40c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9be7d7f497c4d118192c20177a5c455" is_pre_election: true
I20260924 12:50:18.266566 18587 raft_consensus.cc:2454] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20311c138ae745f88ff433a442ba40c3 in term 0.
I20260924 12:50:18.267308 18784 leader_election.cc:302] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20311c138ae745f88ff433a442ba40c3, f9be7d7f497c4d118192c20177a5c455; no voters:
I20260924 12:50:18.267292 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabd801bab4749e883be89ddeca475ca" candidate_uuid: "20311c138ae745f88ff433a442ba40c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066e8d71a11a4065aff4c16a8c3e522a" is_pre_election: true
I20260924 12:50:18.267711 18918 raft_consensus.cc:2781] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:50:18.267704 18719 raft_consensus.cc:2454] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20311c138ae745f88ff433a442ba40c3 in term 0.
I20260924 12:50:18.267879 18918 raft_consensus.cc:482] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:50:18.267978 18918 raft_consensus.cc:3037] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:50:18.270208 18918 raft_consensus.cc:504] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.270787 18918 leader_election.cc:288] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [CANDIDATE]: Term 1 election: Requested vote from peers 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315), f9be7d7f497c4d118192c20177a5c455 (127.2.81.65:45069)
I20260924 12:50:18.271258 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabd801bab4749e883be89ddeca475ca" candidate_uuid: "20311c138ae745f88ff433a442ba40c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066e8d71a11a4065aff4c16a8c3e522a"
I20260924 12:50:18.271294 18587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabd801bab4749e883be89ddeca475ca" candidate_uuid: "20311c138ae745f88ff433a442ba40c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9be7d7f497c4d118192c20177a5c455"
I20260924 12:50:18.271428 18719 raft_consensus.cc:3037] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:50:18.271552 18587 raft_consensus.cc:3037] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:50:18.273730 18719 raft_consensus.cc:2454] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20311c138ae745f88ff433a442ba40c3 in term 1.
I20260924 12:50:18.273849 18587 raft_consensus.cc:2454] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20311c138ae745f88ff433a442ba40c3 in term 1.
I20260924 12:50:18.274190 18785 leader_election.cc:302] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066e8d71a11a4065aff4c16a8c3e522a, 20311c138ae745f88ff433a442ba40c3; no voters:
I20260924 12:50:18.274499 18918 raft_consensus.cc:2781] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:50:18.274964 18918 raft_consensus.cc:686] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 1 LEADER]: Becoming Leader. State: Replica: 20311c138ae745f88ff433a442ba40c3, State: Running, Role: LEADER
I20260924 12:50:18.275435 18918 consensus_queue.cc:237] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } }
I20260924 12:50:18.280217 18446 catalog_manager.cc:5495] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20311c138ae745f88ff433a442ba40c3 (127.2.81.67). New cstate: current_term: 1 leader_uuid: "20311c138ae745f88ff433a442ba40c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20311c138ae745f88ff433a442ba40c3" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 37243 } health_report { overall_health: HEALTHY } } }
W20260924 12:50:18.284793 18896 tablet.cc:2369] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:50:18.315678 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.310628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.2.81.67:37243)
0924 12:50:18.310792 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.310796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.310812 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:50:18.310898 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.310901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.310993 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.311112 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:50:18.311423 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.312133 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.313540 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.313980 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.314085 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.314576 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:50:18.315112 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:50:18.315220 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.315319 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.315394 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 12:50:18.323678 18587 raft_consensus.cc:1261] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Refusing update from remote peer 20311c138ae745f88ff433a442ba40c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:50:18.323704 18719 raft_consensus.cc:1261] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Refusing update from remote peer 20311c138ae745f88ff433a442ba40c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 12:50:18.324596 18765 tablet.cc:2369] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:50:18.324687 18921 consensus_queue.cc:1035] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "066e8d71a11a4065aff4c16a8c3e522a" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:50:18.325098 18918 consensus_queue.cc:1035] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9be7d7f497c4d118192c20177a5c455" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:50:18.336484 18924 mvcc.cc:205] Tried to move back new op lower bound from 7332881278244401152 to 7332881278060441600. Current Snapshot: MvccSnapshot[applied={T|T < 7332881278244401152}]
I20260924 12:50:18.338660 18925 mvcc.cc:205] Tried to move back new op lower bound from 7332881278244401152 to 7332881278060441600. Current Snapshot: MvccSnapshot[applied={T|T < 7332881278244401152}]
I20260924 12:50:18.351243 18445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:50:18.351614 18445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:50:18.355824 18445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Sending DeleteTablet for 3 replicas of tablet dabd801bab4749e883be89ddeca475ca
I20260924 12:50:18.357082 18567 tablet_service.cc:1514] Processing DeleteTablet for tablet dabd801bab4749e883be89ddeca475ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:50:18 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:51322
I20260924 12:50:18.357084 18699 tablet_service.cc:1514] Processing DeleteTablet for tablet dabd801bab4749e883be89ddeca475ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:50:18 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:35830
I20260924 12:50:18.357788 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet dabd801bab4749e883be89ddeca475ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:50:18 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:34876
I20260924 12:50:18.357952 18934 tablet_replica.cc:330] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: stopping tablet replica
I20260924 12:50:18.358392 18934 raft_consensus.cc:2229] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:50:18.358670 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 18484
I20260924 12:50:18.358920 18934 raft_consensus.cc:2258] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:50:18.360160 18936 tablet_replica.cc:330] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: stopping tablet replica
I20260924 12:50:18.360301 18935 tablet_replica.cc:330] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: stopping tablet replica
I20260924 12:50:18.360636 18935 raft_consensus.cc:2229] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:50:18.360677 18936 raft_consensus.cc:2229] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:50:18.361140 18935 raft_consensus.cc:2258] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:50:18.361443 18936 raft_consensus.cc:2258] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:50:18.363183 18935 ts_tablet_manager.cc:1905] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:50:18.363441 18936 ts_tablet_manager.cc:1905] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:50:18.364051 18934 ts_tablet_manager.cc:1905] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:50:18.370213 18935 ts_tablet_manager.cc:1918] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:50:18.370404 18935 log.cc:1196] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/dabd801bab4749e883be89ddeca475ca
I20260924 12:50:18.370991 18935 ts_tablet_manager.cc:1939] T dabd801bab4749e883be89ddeca475ca P f9be7d7f497c4d118192c20177a5c455: Deleting consensus metadata
I20260924 12:50:18.373243 18432 catalog_manager.cc:4841] TS f9be7d7f497c4d118192c20177a5c455 (127.2.81.65:45069): tablet dabd801bab4749e883be89ddeca475ca (table test-table [id=90812385b7ee4d6d81f7f31f105befe2]) successfully deleted
I20260924 12:50:18.375855 18934 ts_tablet_manager.cc:1918] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:50:18.376086 18934 log.cc:1196] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/dabd801bab4749e883be89ddeca475ca
I20260924 12:50:18.376711 18934 ts_tablet_manager.cc:1939] T dabd801bab4749e883be89ddeca475ca P 066e8d71a11a4065aff4c16a8c3e522a: Deleting consensus metadata
I20260924 12:50:18.377035 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.359772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.2.81.65:45069)
0924 12:50:18.359917 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.359923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.359943 (+ 20us) server_negotiation.cc:407] Connection header received
0924 12:50:18.360001 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.360006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.360111 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.360227 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:18.361986 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.363065 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.364890 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.365567 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.365701 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.369128 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:18.369143 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:18.369163 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:18.369204 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:18.373249 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.374154 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.374165 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.374168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.374331 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.374753 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.374758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.374762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.375226 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:18.375520 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.376281 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.376565 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 12:50:18.377646 18627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:18.377923 18936 ts_tablet_manager.cc:1918] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:50:18.378187 18936 log.cc:1196] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/dabd801bab4749e883be89ddeca475ca
I20260924 12:50:18.378806 18936 ts_tablet_manager.cc:1939] T dabd801bab4749e883be89ddeca475ca P 20311c138ae745f88ff433a442ba40c3: Deleting consensus metadata
I20260924 12:50:18.378894 18433 catalog_manager.cc:4841] TS 066e8d71a11a4065aff4c16a8c3e522a (127.2.81.66:41315): tablet dabd801bab4749e883be89ddeca475ca (table test-table [id=90812385b7ee4d6d81f7f31f105befe2]) successfully deleted
I20260924 12:50:18.380934 18434 catalog_manager.cc:4841] TS 20311c138ae745f88ff433a442ba40c3 (127.2.81.67:37243): tablet dabd801bab4749e883be89ddeca475ca (table test-table [id=90812385b7ee4d6d81f7f31f105befe2]) successfully deleted
I20260924 12:50:18.500517 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 18484
I20260924 12:50:18.520998 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 18636
I20260924 12:50:18.535223 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.521749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.2.81.66:41315)
0924 12:50:18.521952 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.521958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.521980 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:50:18.522145 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.522150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.522215 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.522307 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:18.523607 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.524331 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.525733 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.526135 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.526250 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.529063 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:18.529078 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:18.529094 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:18.529133 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:18.532014 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.532831 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.532839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.532841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.533128 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.533655 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.533660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.533664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.534059 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:18.534352 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.534658 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.534874 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:50:18.535902 18759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:18.629675 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 18636
I20260924 12:50:18.650058 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 18767
I20260924 12:50:18.666301 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:18.650846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.2.81.67:37243)
0924 12:50:18.651035 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:18.651041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:18.651089 (+ 48us) server_negotiation.cc:407] Connection header received
0924 12:50:18.651281 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:18.651285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:18.651361 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:18.651468 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:18.652785 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.653682 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.655586 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:18.656160 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:18.656278 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:18.659255 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:18.659270 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:18.659286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:18.659331 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:18.663135 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.663996 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.664005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.664007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.664242 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:18.664629 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:18.664633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:18.664635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:18.665089 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:18.665373 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:18.665681 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 12:50:18.665901 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 12:50:18.667101 18890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:18.791517 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 18767
I20260924 12:50:18.806994 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 18414
I20260924 12:50:18.808264 18477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:18.927400 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 18414
I20260924 12:50:38.945438 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45209
--webserver_interface=127.2.81.126
--webserver_port=39341
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:50:39.287331 18941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:39.287679 18941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:39.287818 18941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:39.296314 18941 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 12:50:39.296445 18941 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 12:50:39.296502 18941 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 12:50:39.296578 18941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:50:39.296653 18941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:39.296748 18941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:39.296810 18941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:50:39.296883 18941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:50:39.308723 18941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=39341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:39.310163 18941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:39.311571 18941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:39.323756 18947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:39.323975 18946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:39.323961 18949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:39.324589 18941 server_base.cc:1034] running on GCE node
I20260924 12:50:39.325521 18941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:39.327086 18941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:39.328367 18941 hybrid_clock.cc:648] HybridClock initialized: now 1790254239328293 us; error 35 us; skew 500 ppm
I20260924 12:50:39.328799 18941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:39 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254239, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:39.337325 18941 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:50:39.339026 18941 webserver.cc:460] Webserver started at http://127.2.81.126:39341/ using document root <none> and password file <none>
I20260924 12:50:39.339771 18941 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:39.339879 18941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:39.345834 18941 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 12:50:39.349068 18956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:39.350576 18941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:50:39.350740 18941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "b999ae8319154608b62dffa05c9d82f9"
format_stamp: "Formatted at 2026-09-24 12:49:53 on dist-test-slave-9tm1"
server_key: "12ac9b933c451b2bbdb09efb59342997"
server_key_iv: "e817d4a5d20483e99ed774388697e723"
server_key_version: "encryptionkey@0"
I20260924 12:50:39.351727 18941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:39.378755 18941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:39.383474 18941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:39.383752 18941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:39.403168 18941 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45209
I20260924 12:50:39.403224 19007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45209 every 8 connection(s)
I20260924 12:50:39.404697 18941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:50:39.410321 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 18941
I20260924 12:50:39.411377 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:45069
--local_ip_for_outbound_sockets=127.2.81.65
--tserver_master_addrs=127.2.81.126:45209
--webserver_port=36031
--webserver_interface=127.2.81.65
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:50:39.412559 19008 sys_catalog.cc:263] Verifying existing consensus state
I20260924 12:50:39.417263 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap starting.
I20260924 12:50:39.439256 19008 log.cc:824] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Log is configured to *not* fsync() on all Append() calls
I20260924 12:50:39.458724 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 12:50:39.459321 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Bootstrap complete.
I20260924 12:50:39.469126 19008 raft_consensus.cc:348] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:39.470546 19008 raft_consensus.cc:729] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b999ae8319154608b62dffa05c9d82f9, State: Initialized, Role: FOLLOWER
I20260924 12:50:39.471258 19008 consensus_queue.cc:260] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:39.471894 19008 raft_consensus.cc:388] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:50:39.472054 19008 raft_consensus.cc:482] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:50:39.472252 19008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 12:50:39.477097 19008 raft_consensus.cc:504] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:39.477694 19008 leader_election.cc:302] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b999ae8319154608b62dffa05c9d82f9; no voters:
I20260924 12:50:39.478392 19008 leader_election.cc:288] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 12:50:39.478550 19012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 12:50:39.479799 19012 raft_consensus.cc:686] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [term 3 LEADER]: Becoming Leader. State: Replica: b999ae8319154608b62dffa05c9d82f9, State: Running, Role: LEADER
I20260924 12:50:39.480491 19012 consensus_queue.cc:237] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } }
I20260924 12:50:39.480760 19008 sys_catalog.cc:564] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:50:39.483129 19013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b999ae8319154608b62dffa05c9d82f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } } }
I20260924 12:50:39.483563 19013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:50:39.485832 19014 sys_catalog.cc:455] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b999ae8319154608b62dffa05c9d82f9. Latest consensus state: current_term: 3 leader_uuid: "b999ae8319154608b62dffa05c9d82f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b999ae8319154608b62dffa05c9d82f9" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45209 } } }
I20260924 12:50:39.486218 19014 sys_catalog.cc:458] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9 [sys.catalog]: This master's current role is: LEADER
I20260924 12:50:39.487812 19017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:50:39.496675 19017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:50:39.498597 19017 catalog_manager.cc:1259] Loaded cluster ID: 5f2252a736e94ae084748f63738e0e50
I20260924 12:50:39.498795 19017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:50:39.503266 19017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:50:39.505430 19017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Loaded TSK: 0
I20260924 12:50:39.505656 19017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b999ae8319154608b62dffa05c9d82f9: Loaded TSK: 1
F20260924 12:50:39.508085 19017 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03081759ad google::LogMessage::Fail() at ??:0
@ 0x7f030817888a google::LogMessage::SendToLog() at ??:0
@ 0x7f03081754de google::LogMessage::Flush() at ??:0
@ 0x7f0308176a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f030bed97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f030bec873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f031dba81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f031dba7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f031dc332a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f031dc330c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f031dc7d5e9 std::function<>::operator()() at ??:0
@ 0x7f031dbabcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f031dba9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f031dc31610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f031dc31439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f030adc1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f030adc491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f030adc4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f030ad9feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f030e0796db start_thread at ??:0
@ 0x7f030700f61f clone at ??:0
W20260924 12:50:39.840562 19010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:39.840884 19010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:39.841064 19010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:39.848680 19010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:39.848826 19010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:39.849020 19010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:50:39.859361 19010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:45069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=36031
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:39.860687 19010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:39.861986 19010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:39.874851 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:39.874930 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:39.874955 19058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:39.875661 19010 server_base.cc:1034] running on GCE node
I20260924 12:50:39.876300 19010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:39.877349 19010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:39.878633 19010 hybrid_clock.cc:648] HybridClock initialized: now 1790254239878586 us; error 37 us; skew 500 ppm
I20260924 12:50:39.878990 19010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:39 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254239, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:39.882525 19010 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:50:39.884213 19010 webserver.cc:460] Webserver started at http://127.2.81.65:36031/ using document root <none> and password file <none>
I20260924 12:50:39.884953 19010 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:39.885120 19010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:39.891167 19010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:50:39.894485 19065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:39.895979 19010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:50:39.896149 19010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "f9be7d7f497c4d118192c20177a5c455"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "295fb04499c834161d0a6a56f37f9db2"
server_key_iv: "659b38f6d14e11b0b38120421e3b3858"
server_key_version: "encryptionkey@0"
I20260924 12:50:39.897291 19010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:39.912810 19010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:39.917385 19010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:39.917640 19010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:39.918601 19010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:39.920002 19010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:39.920099 19010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:39.920182 19010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:39.920234 19010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:39.952764 19010 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:45069
I20260924 12:50:39.952844 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:45069 every 8 connection(s)
I20260924 12:50:39.954363 19010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:50:39.959770 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:39.957699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:34977)
0924 12:50:39.958486 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:39.958825 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":308,"threads_started":1}
I20260924 12:50:39.960089 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19010
W20260924 12:50:39.960597 19178 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45209 (0 consecutive failures): Network error: Failed to ping master at 127.2.81.126:45209: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
I20260924 12:50:39.960891 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:41315
--local_ip_for_outbound_sockets=127.2.81.66
--tserver_master_addrs=127.2.81.126:45209
--webserver_port=45371
--webserver_interface=127.2.81.66
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:50:39.961712 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:39.961314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:55537)
0924 12:50:39.961511 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:39.961588 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 12:50:39.962841 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:39.962404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:58451)
0924 12:50:39.962616 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:39.962708 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 12:50:39.963105 19178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 12:50:40.382833 19181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:40.383133 19181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:40.383227 19181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:40.390872 19181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:40.391000 19181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:40.391114 19181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:50:40.402001 19181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:41315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=45371
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:40.403318 19181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:40.404595 19181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:40.417984 19186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:40.418053 19187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:40.418079 19189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:40.418696 19181 server_base.cc:1034] running on GCE node
I20260924 12:50:40.419344 19181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:40.420410 19181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:40.421658 19181 hybrid_clock.cc:648] HybridClock initialized: now 1790254240421587 us; error 72 us; skew 500 ppm
I20260924 12:50:40.422008 19181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:40 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:40.425246 19181 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:50:40.426909 19181 webserver.cc:460] Webserver started at http://127.2.81.66:45371/ using document root <none> and password file <none>
I20260924 12:50:40.427603 19181 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:40.427713 19181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:40.433223 19181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:50:40.436111 19196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:40.437636 19181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:50:40.437790 19181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "066e8d71a11a4065aff4c16a8c3e522a"
format_stamp: "Formatted at 2026-09-24 12:49:54 on dist-test-slave-9tm1"
server_key: "44b177374b96b8a7e6d53d539bd7130c"
server_key_iv: "029cc210f97036d5f1a7d12f097cc9f9"
server_key_version: "encryptionkey@0"
I20260924 12:50:40.438812 19181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:40.466898 19181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:40.471544 19181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:40.471809 19181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:40.472848 19181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:40.474411 19181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:40.474517 19181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:40.474606 19181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:40.474671 19181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:40.512346 19181 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:41315
I20260924 12:50:40.512408 19308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:41315 every 8 connection(s)
I20260924 12:50:40.513886 19181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:50:40.518891 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.516934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:49019)
0924 12:50:40.517733 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.518000 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":307,"threads_started":1}
I20260924 12:50:40.519084 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19181
W20260924 12:50:40.519642 19309 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45209 (0 consecutive failures): Network error: Failed to ping master at 127.2.81.126:45209: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
I20260924 12:50:40.519881 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:37243
--local_ip_for_outbound_sockets=127.2.81.67
--tserver_master_addrs=127.2.81.126:45209
--webserver_port=32807
--webserver_interface=127.2.81.67
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:50:40.520576 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.520206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:32919)
0924 12:50:40.520378 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.520457 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 12:50:40.521516 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.521128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:55007)
0924 12:50:40.521287 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.521373 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 12:50:40.521761 19309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 12:50:40.859474 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:40.859769 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:40.859864 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:40.867234 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:40.867360 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:40.867484 19312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:50:40.877664 19312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:37243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=32807
--tserver_master_addrs=127.2.81.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:40.878885 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:40.880093 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:40.893246 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:40.893199 19318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:40.893209 19320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:40.893885 19312 server_base.cc:1034] running on GCE node
I20260924 12:50:40.894508 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:40.895488 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:40.896726 19312 hybrid_clock.cc:648] HybridClock initialized: now 1790254240896658 us; error 45 us; skew 500 ppm
I20260924 12:50:40.897114 19312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:40 dist-test-slave-9tm1 krb5kdc[17863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:40.900099 19312 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:50:40.901623 19312 webserver.cc:460] Webserver started at http://127.2.81.67:32807/ using document root <none> and password file <none>
I20260924 12:50:40.902276 19312 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:40.902375 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:40.907725 19312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 12:50:40.910706 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:40.912256 19312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:50:40.912431 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "20311c138ae745f88ff433a442ba40c3"
format_stamp: "Formatted at 2026-09-24 12:49:55 on dist-test-slave-9tm1"
server_key: "e3517a2e0152bfb7af6b4d2cad4a0494"
server_key_iv: "6d51c8c9b8fca1f3b2cac2b15736afba"
server_key_version: "encryptionkey@0"
I20260924 12:50:40.913489 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:40.931097 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:40.936450 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:40.936704 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:40.937798 19312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:40.939129 19312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:40.939219 19312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:40.939327 19312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:40.939393 19312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:40.964561 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.963739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:45677)
0924 12:50:40.964216 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.964332 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":225,"threads_started":1}
I20260924 12:50:40.972011 19312 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:37243
I20260924 12:50:40.972081 19440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:37243 every 8 connection(s)
I20260924 12:50:40.973552 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:50:40.978401 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.976547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:43907)
0924 12:50:40.977333 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.977538 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":296,"threads_started":1}
W20260924 12:50:40.979123 19441 heartbeater.cc:644] Failed to heartbeat to 127.2.81.126:45209 (0 consecutive failures): Network error: Failed to ping master at 127.2.81.126:45209: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
I20260924 12:50:40.979975 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.979648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:37849)
0924 12:50:40.979796 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.979873 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 12:50:40.980839 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:40.980440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:34275)
0924 12:50:40.980641 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:40.980730 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260924 12:50:40.981097 19441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 12:50:40.983582 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19312
I20260924 12:50:41.523257 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:41.522376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:37015)
0924 12:50:41.522897 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:41.523017 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":262,"threads_started":1}
I20260924 12:50:41.966391 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:41.965609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:40285)
0924 12:50:41.966111 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:41.966203 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
I20260924 12:50:41.982538 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:41.981691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:60915)
0924 12:50:41.982195 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:41.982302 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260924 12:50:42.525025 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:42.524249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:48911)
0924 12:50:42.524736 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:42.524824 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
I20260924 12:50:42.968305 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:42.967514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:34249)
0924 12:50:42.968038 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:42.968129 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":305,"threads_started":1}
I20260924 12:50:42.984508 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:42.983691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:52369)
0924 12:50:42.984235 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:42.984329 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":265,"threads_started":1}
I20260924 12:50:43.526686 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:43.525942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:60005)
0924 12:50:43.526411 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:43.526498 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":240,"threads_started":1}
I20260924 12:50:43.970605 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:43.969585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:58363)
0924 12:50:43.970231 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:43.970343 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":232,"threads_started":1}
I20260924 12:50:43.986604 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:43.985737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:57131)
0924 12:50:43.986304 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:43.986418 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
I20260924 12:50:44.528570 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:44.527774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:54267)
0924 12:50:44.528276 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:44.528370 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260924 12:50:44.972772 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:44.971735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:35727)
0924 12:50:44.972452 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:44.972569 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":351,"threads_started":1}
I20260924 12:50:44.988862 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:44.987697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:45167)
0924 12:50:44.988332 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:44.988447 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":396,"threads_started":1}
I20260924 12:50:45.530303 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:45.529548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:33913)
0924 12:50:45.530038 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:45.530127 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260924 12:50:45.974625 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:45.973779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:42491)
0924 12:50:45.974366 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:45.974462 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":268,"threads_started":1}
I20260924 12:50:45.990767 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:45.989928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:41631)
0924 12:50:45.990462 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:45.990590 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
I20260924 12:50:46.532591 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:46.531627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:46315)
0924 12:50:46.532224 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:46.532332 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":278,"threads_started":1}
I20260924 12:50:46.976470 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:46.975598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:47883)
0924 12:50:46.976137 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:46.976238 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":243,"threads_started":1}
I20260924 12:50:46.992635 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:46.991771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:54049)
0924 12:50:46.992339 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:46.992447 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":313,"threads_started":1}
I20260924 12:50:47.534740 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:47.533878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:59505)
0924 12:50:47.534428 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:47.534528 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":233,"threads_started":1}
I20260924 12:50:47.978394 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:47.977663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:36279)
0924 12:50:47.978135 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:47.978228 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":230,"threads_started":1}
I20260924 12:50:47.994419 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:47.993548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:52505)
0924 12:50:47.994108 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:47.994195 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":281,"threads_started":1}
I20260924 12:50:48.536334 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:48.535643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:49169)
0924 12:50:48.536068 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:48.536173 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260924 12:50:48.980136 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:48.979344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:51109)
0924 12:50:48.979853 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:48.979939 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
I20260924 12:50:48.996419 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:48.995567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:46539)
0924 12:50:48.996163 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:48.996252 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":376,"threads_started":1}
I20260924 12:50:49.538152 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:49.537360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:42369)
0924 12:50:49.537872 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:49.537963 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":260,"threads_started":1}
I20260924 12:50:49.981993 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:49.981129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:38479)
0924 12:50:49.981616 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:49.981815 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260924 12:50:49.998199 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:49.997437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:34859)
0924 12:50:49.997926 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:49.998022 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":246,"threads_started":1}
I20260924 12:50:50.540060 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:50.539232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:60739)
0924 12:50:50.539760 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:50.539863 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":300,"threads_started":1}
I20260924 12:50:50.983826 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:50.983048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:52351)
0924 12:50:50.983526 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:50.983629 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":239,"threads_started":1}
I20260924 12:50:50.999881 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:50.999158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:46219)
0924 12:50:50.999631 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:50.999722 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260924 12:50:51.541963 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:51.541131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:40463)
0924 12:50:51.541590 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:51.541700 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260924 12:50:51.985711 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:51.984882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:56275)
0924 12:50:51.985450 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:51.985546 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260924 12:50:52.001775 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:52.000910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:55401)
0924 12:50:52.001456 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:52.001590 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":212,"threads_started":1}
I20260924 12:50:52.543892 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:52.543020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:51279)
0924 12:50:52.543582 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:52.543675 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
I20260924 12:50:52.987480 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:52.986701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:35825)
0924 12:50:52.987198 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:52.987297 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":220,"threads_started":1}
I20260924 12:50:53.003466 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:53.002685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:53607)
0924 12:50:53.003208 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:53.003295 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":276,"threads_started":1}
I20260924 12:50:53.545760 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:53.544840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:39605)
0924 12:50:53.545351 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:53.545468 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260924 12:50:53.989378 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:53.988606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:41637)
0924 12:50:53.989114 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:53.989197 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260924 12:50:54.005227 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:54.004366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:55253)
0924 12:50:54.004905 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:54.005039 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":297,"threads_started":1}
I20260924 12:50:54.547431 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:54.546700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:41569)
0924 12:50:54.547149 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:54.547238 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":219,"threads_started":1}
I20260924 12:50:54.991443 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:54.990657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:54515)
0924 12:50:54.991157 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:54.991261 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260924 12:50:55.006947 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:55.006217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:34479)
0924 12:50:55.006676 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:55.006764 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260924 12:50:55.549410 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:55.548560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.66:45175)
0924 12:50:55.549109 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:55.549228 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":224,"threads_started":1}
I20260924 12:50:55.989295 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19010
I20260924 12:50:55.994346 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:55.992699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.65:53869)
0924 12:50:55.993802 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:55.994048 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":419,"threads_started":1}
I20260924 12:50:56.009287 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:56.008310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45209 (local address 127.2.81.67:50305)
0924 12:50:56.008957 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:56.009099 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.126:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":334,"threads_started":1}
I20260924 12:50:56.011327 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:55.990429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.2.81.65:45069)
0924 12:50:55.991360 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:55.991380 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:55.991431 (+ 51us) server_negotiation.cc:407] Connection header received
0924 12:50:55.994679 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:55.994689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:55.995160 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:55.995519 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:55.997125 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:55.998437 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:55.999820 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:56.000578 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:56.000766 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:56.003615 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:56.003625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:56.003643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:56.003693 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:56.007108 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:56.008360 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:56.008369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:56.008377 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:56.008621 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:56.009073 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:56.009078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:56.009080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:56.009764 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:56.010089 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:56.010575 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 12:50:56.011036 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":528,"threads_started":1}
I20260924 12:50:56.012952 19173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:56.100410 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19010
I20260924 12:50:56.114265 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19181
I20260924 12:50:56.133533 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:56.115211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.2.81.66:41315)
0924 12:50:56.115724 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:56.115738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:56.115775 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:50:56.117424 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:56.117443 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:56.117886 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:56.118226 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:56.119675 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:56.121176 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:56.122528 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:56.123278 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:56.123473 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:56.126374 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:56.126382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:56.126407 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:56.126460 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:56.129530 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:56.130348 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:56.130356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:56.130363 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:56.130567 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:56.130957 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:56.130961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:56.130963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:56.131523 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:56.131832 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:56.132288 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 12:50:56.132763 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260924 12:50:56.135178 19304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:56.220871 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19181
I20260924 12:50:56.235179 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19312
I20260924 12:50:56.253906 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:56.236254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.2.81.67:37243)
0924 12:50:56.236899 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:56.236914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:56.236959 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:50:56.238720 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:56.238732 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:56.239194 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:56.239561 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:56.241062 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:56.242317 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:56.243588 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:56.244281 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:56.244473 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:56.247236 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:56.247244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:56.247262 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:56.247316 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:56.250482 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:56.251242 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:56.251249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:56.251256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:56.251435 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:56.251805 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:56.251809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:56.251811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:56.252356 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:56.252655 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:56.253099 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 12:50:56.253590 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":325,"threads_started":1}
I20260924 12:50:56.255213 19436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:50:56.340781 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19312
2026-09-24T12:50:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64477 ms)
[----------] 30 tests from SecurityITest (306616 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:50:56 dist-test-slave-9tm1 krb5kdc[19498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:50:56 dist-test-slave-9tm1 krb5kdc[19498](info): set up 2 sockets
Sep 24 12:50:56 dist-test-slave-9tm1 krb5kdc[19498](info): commencing operation
krb5kdc: starting...
W20260924 12:50:57.393116 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:50:57 dist-test-slave-9tm1 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:50:57.646919 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37003
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:37003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:50:57.988962 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:57.989351 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:57.989490 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:57.997344 19511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:50:57.997505 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:57.997589 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:57.997666 19511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:50:57.997727 19511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:50:58.008749 19511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:37003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:58.010195 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:58.011552 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:58.024212 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:58.024106 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:58.024255 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:58.024742 19511 server_base.cc:1034] running on GCE node
I20260924 12:50:58.025688 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:58.027257 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:58.028477 19511 hybrid_clock.cc:648] HybridClock initialized: now 1790254258028424 us; error 55 us; skew 500 ppm
I20260924 12:50:58.028855 19511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:58 dist-test-slave-9tm1 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254258, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:58.032167 19511 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:50:58.033874 19511 webserver.cc:460] Webserver started at http://127.2.81.126:40481/ using document root <none> and password file <none>
I20260924 12:50:58.034569 19511 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:58.034686 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:58.034978 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:50:58.038100 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "737d82cbc30b4a15937a57b99c8f360e"
format_stamp: "Formatted at 2026-09-24 12:50:58 on dist-test-slave-9tm1"
server_key: "ee5258c01481c841cc3ffe4b88edf6d0"
server_key_iv: "fa7bdec96156c486fbc7cd9de968d81f"
server_key_version: "encryptionkey@0"
I20260924 12:50:58.038856 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "737d82cbc30b4a15937a57b99c8f360e"
format_stamp: "Formatted at 2026-09-24 12:50:58 on dist-test-slave-9tm1"
server_key: "ee5258c01481c841cc3ffe4b88edf6d0"
server_key_iv: "fa7bdec96156c486fbc7cd9de968d81f"
server_key_version: "encryptionkey@0"
I20260924 12:50:58.044436 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:50:58.048521 19526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:58.050133 19511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:50:58.050313 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "737d82cbc30b4a15937a57b99c8f360e"
format_stamp: "Formatted at 2026-09-24 12:50:58 on dist-test-slave-9tm1"
server_key: "ee5258c01481c841cc3ffe4b88edf6d0"
server_key_iv: "fa7bdec96156c486fbc7cd9de968d81f"
server_key_version: "encryptionkey@0"
I20260924 12:50:58.050540 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:58.069444 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:58.074323 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:58.074651 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:58.096091 19511 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:37003
I20260924 12:50:58.096143 19577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:37003 every 8 connection(s)
I20260924 12:50:58.097601 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:50:58.101522 19578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:50:58.102234 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19511
I20260924 12:50:58.102473 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:50:58.103309 2373 external_mini_cluster.cc:1428] Setting key c47872ea3eabe26be615d461a2c7dcfa
I20260924 12:50:58.117221 19578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e: Bootstrap starting.
Sep 24 12:50:58 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:58.121423 19578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e: Neither blocks nor log segments found. Creating new log.
I20260924 12:50:58.122792 19578 log.cc:824] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:50:58.126549 19578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e: No bootstrap required, opened a new log
I20260924 12:50:58.129828 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:58.105712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.2.81.126:37003)
0924 12:50:58.106582 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:58.106596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:58.106645 (+ 49us) server_negotiation.cc:407] Connection header received
0924 12:50:58.108647 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:58.108679 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:58.109284 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:58.109708 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:58.111608 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:58.113000 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:58.114898 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:58.115650 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:58.115790 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:58.121130 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:58.121141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:58.121158 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:58.121206 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:58.125128 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:58.126118 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:58.126125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:58.126132 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:58.126317 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:58.126801 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:58.126807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:58.126810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:58.127393 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:58.127652 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:58.128179 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 12:50:58.128621 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":355,"threads_started":1}
I20260924 12:50:58.135366 19578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d82cbc30b4a15937a57b99c8f360e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37003 } }
I20260924 12:50:58.135771 19578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:50:58.135869 19578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737d82cbc30b4a15937a57b99c8f360e, State: Initialized, Role: FOLLOWER
I20260924 12:50:58.136471 19578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d82cbc30b4a15937a57b99c8f360e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37003 } }
I20260924 12:50:58.136713 19578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:50:58.136842 19578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:50:58.137035 19578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:50:58.139325 19578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d82cbc30b4a15937a57b99c8f360e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37003 } }
I20260924 12:50:58.139906 19578 leader_election.cc:302] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 737d82cbc30b4a15937a57b99c8f360e; no voters:
I20260924 12:50:58.140506 19578 leader_election.cc:288] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:50:58.140647 19583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:50:58.141292 19583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [term 1 LEADER]: Becoming Leader. State: Replica: 737d82cbc30b4a15937a57b99c8f360e, State: Running, Role: LEADER
I20260924 12:50:58.141872 19583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d82cbc30b4a15937a57b99c8f360e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37003 } }
I20260924 12:50:58.142531 19578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:50:58.144476 19585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737d82cbc30b4a15937a57b99c8f360e. Latest consensus state: current_term: 1 leader_uuid: "737d82cbc30b4a15937a57b99c8f360e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d82cbc30b4a15937a57b99c8f360e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37003 } } }
I20260924 12:50:58.144531 19584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737d82cbc30b4a15937a57b99c8f360e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737d82cbc30b4a15937a57b99c8f360e" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37003 } } }
I20260924 12:50:58.144997 19585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [sys.catalog]: This master's current role is: LEADER
I20260924 12:50:58.145001 19584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e [sys.catalog]: This master's current role is: LEADER
I20260924 12:50:58.147429 19588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:50:58.153893 19588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:50:58.165313 19588 catalog_manager.cc:1347] Generated new cluster ID: 7b475c3206a244dfa5cb175ff3ec45b4
I20260924 12:50:58.165437 19588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:50:58.189455 19588 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:50:58.191133 19588 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:50:58.200549 19588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e: Generated new TSK 0
I20260924 12:50:58.201583 19588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:50:58.274906 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37003
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:50:58.658174 19604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:58.658471 19604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:58.658597 19604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:58.666590 19604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:58.666733 19604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:58.666887 19604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:50:58.678124 19604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:58.679484 19604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:58.680809 19604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:58.694491 19612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:58.694444 19609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:58.694715 19610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:58.696185 19604 server_base.cc:1034] running on GCE node
I20260924 12:50:58.697132 19604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:58.698611 19604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:58.699883 19604 hybrid_clock.cc:648] HybridClock initialized: now 1790254258699818 us; error 66 us; skew 500 ppm
I20260924 12:50:58.700254 19604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:58 dist-test-slave-9tm1 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254258, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:58.707890 19604 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:50:58.709548 19604 webserver.cc:460] Webserver started at http://127.2.81.65:43097/ using document root <none> and password file <none>
I20260924 12:50:58.710282 19604 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:58.710410 19604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:58.710732 19604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:50:58.713963 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "aebf579d532743628c9d7ec8a48fdc7a"
format_stamp: "Formatted at 2026-09-24 12:50:58 on dist-test-slave-9tm1"
server_key: "670878e3bd7f19ac9163b6ad4df21c22"
server_key_iv: "5ad821eadac4c1750d52c9b335cb324c"
server_key_version: "encryptionkey@0"
I20260924 12:50:58.714886 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "aebf579d532743628c9d7ec8a48fdc7a"
format_stamp: "Formatted at 2026-09-24 12:50:58 on dist-test-slave-9tm1"
server_key: "670878e3bd7f19ac9163b6ad4df21c22"
server_key_iv: "5ad821eadac4c1750d52c9b335cb324c"
server_key_version: "encryptionkey@0"
I20260924 12:50:58.720692 19604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:50:58.725117 19619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:58.726882 19604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:50:58.727102 19604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "aebf579d532743628c9d7ec8a48fdc7a"
format_stamp: "Formatted at 2026-09-24 12:50:58 on dist-test-slave-9tm1"
server_key: "670878e3bd7f19ac9163b6ad4df21c22"
server_key_iv: "5ad821eadac4c1750d52c9b335cb324c"
server_key_version: "encryptionkey@0"
I20260924 12:50:58.727326 19604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:58.757339 19604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:58.762833 19604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:58.763137 19604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:58.764204 19604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:58.765713 19604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:58.765798 19604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:58.765915 19604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:58.765967 19604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:58.797878 19604 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:45591
I20260924 12:50:58.797946 19731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:45591 every 8 connection(s)
I20260924 12:50:58.799152 19604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:50:58.807816 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19604
I20260924 12:50:58.808044 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:50:58.808686 2373 external_mini_cluster.cc:1428] Setting key 4d2252c997553386bb499c8767d83608
Sep 24 12:50:58 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254258, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:58.824404 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:58.801882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:41305 (local address 127.2.81.126:37003)
0924 12:50:58.802345 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:58.802350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:58.803251 (+ 901us) server_negotiation.cc:407] Connection header received
0924 12:50:58.805425 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:58.805431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:58.805536 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:58.805646 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:58.808092 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:58.809060 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:58.811051 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:58.811576 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:58.811691 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:58.816344 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:58.816351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:58.816359 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:58.816398 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:58.820186 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:58.821184 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:58.821191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:58.821195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:58.821392 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:58.821922 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:58.821926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:58.821929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:58.822297 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:58.822455 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:58.823783 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0924 12:50:58.824043 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260924 12:50:58.825879 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:58.802320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37003 (local address 127.2.81.65:41305)
0924 12:50:58.803065 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:58.803106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:58.804902 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:58.805926 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:58.805935 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:58.806523 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:50:58.807702 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:58.807742 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:58.809268 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:58.809275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:58.810798 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:58.810819 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:58.811866 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:58.811875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:58.812218 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:58.813038 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:50:58.813073 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:50:58.815977 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:50:58.820407 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:58.820419 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:58.820441 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:58.821027 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:58.821568 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:58.821573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:58.821576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:58.821796 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:58.822568 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:50:58.822579 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:50:58.823216 (+ 637us) client_negotiation.cc:769] Sending connection context
0924 12:50:58.823620 (+ 404us) client_negotiation.cc:240] Negotiation successful
0924 12:50:58.824250 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":288,"threads_started":1}
I20260924 12:50:58.829047 19732 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37003
I20260924 12:50:58.829694 19732 heartbeater.cc:464] Registering TS with master...
I20260924 12:50:58.831059 19732 heartbeater.cc:511] Master 127.2.81.126:37003 requested a full tablet report, sending...
I20260924 12:50:58.834233 19543 ts_manager.cc:194] Registered new tserver with Master: aebf579d532743628c9d7ec8a48fdc7a (127.2.81.65:45591)
I20260924 12:50:58.837309 19543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:41305
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:50:58.878703 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37003
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:50:59.240401 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:59.240720 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:59.240820 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:59.248826 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:59.248967 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:59.249132 19738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:50:59.260058 19738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:59.261708 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:59.263628 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:59.281440 19743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:59.281541 19744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:59.281493 19746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:59.283097 19738 server_base.cc:1034] running on GCE node
I20260924 12:50:59.283972 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:59.285303 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:59.286662 19738 hybrid_clock.cc:648] HybridClock initialized: now 1790254259286591 us; error 64 us; skew 500 ppm
I20260924 12:50:59.287057 19738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:50:59 dist-test-slave-9tm1 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:59.291009 19738 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:50:59.292794 19738 webserver.cc:460] Webserver started at http://127.2.81.66:38585/ using document root <none> and password file <none>
I20260924 12:50:59.293632 19738 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:59.293769 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:59.294094 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:50:59.297354 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "1dd5fd482fbe43579a6db44a329f1a68"
format_stamp: "Formatted at 2026-09-24 12:50:59 on dist-test-slave-9tm1"
server_key: "fdc1e684fc87368dd70794c5100a837d"
server_key_iv: "6de75220abe298c3d413bd0dcd18832d"
server_key_version: "encryptionkey@0"
I20260924 12:50:59.298251 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "1dd5fd482fbe43579a6db44a329f1a68"
format_stamp: "Formatted at 2026-09-24 12:50:59 on dist-test-slave-9tm1"
server_key: "fdc1e684fc87368dd70794c5100a837d"
server_key_iv: "6de75220abe298c3d413bd0dcd18832d"
server_key_version: "encryptionkey@0"
I20260924 12:50:59.305225 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 12:50:59.310032 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:59.311780 19738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:50:59.311978 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "1dd5fd482fbe43579a6db44a329f1a68"
format_stamp: "Formatted at 2026-09-24 12:50:59 on dist-test-slave-9tm1"
server_key: "fdc1e684fc87368dd70794c5100a837d"
server_key_iv: "6de75220abe298c3d413bd0dcd18832d"
server_key_version: "encryptionkey@0"
I20260924 12:50:59.312206 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:59.332113 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:59.336766 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:59.337105 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:59.338263 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:59.339802 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:59.339883 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:59.340029 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:59.340082 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:59.377921 19738 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:35975
I20260924 12:50:59.377915 19865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:35975 every 8 connection(s)
I20260924 12:50:59.380077 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:50:59.386735 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19738
I20260924 12:50:59.386941 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:50:59.387398 2373 external_mini_cluster.cc:1428] Setting key d7ebccaed6ad1ca7fd2dbeef3a20a957
Sep 24 12:50:59 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:59.405325 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:59.383859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:47925 (local address 127.2.81.126:37003)
0924 12:50:59.384402 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:59.384407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:59.386414 (+ 2007us) server_negotiation.cc:407] Connection header received
0924 12:50:59.388618 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:59.388624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:59.388717 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:59.388823 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:59.390791 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.391587 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.393467 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.393951 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.394083 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:59.397965 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:59.397972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:59.397977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:59.398021 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:59.401737 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:59.402623 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:59.402630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:59.402633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:59.402900 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:59.403377 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:59.403382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:59.403385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:59.403760 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:59.403917 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:59.404681 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 12:50:59.404926 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260924 12:50:59.406388 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:59.384615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37003 (local address 127.2.81.66:47925)
0924 12:50:59.386183 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:59.386253 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:59.388304 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:59.389156 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:59.389166 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:59.389906 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:50:59.390576 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:59.390598 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.391766 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.391770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:59.393273 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:59.393291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.394218 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.394226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:59.394481 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:59.395194 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:50:59.395224 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:50:59.397632 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:50:59.401945 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:59.401954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:59.401970 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:59.402484 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:59.403040 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:59.403044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:59.403048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:59.403258 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:59.403901 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:50:59.403909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:50:59.404302 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 12:50:59.404582 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 12:50:59.405175 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":468,"threads_started":1}
I20260924 12:50:59.408249 19866 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37003
I20260924 12:50:59.408560 19866 heartbeater.cc:464] Registering TS with master...
I20260924 12:50:59.409403 19866 heartbeater.cc:511] Master 127.2.81.126:37003 requested a full tablet report, sending...
I20260924 12:50:59.411032 19543 ts_manager.cc:194] Registered new tserver with Master: 1dd5fd482fbe43579a6db44a329f1a68 (127.2.81.66:35975)
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
I20260924 12:50:59.411912 19543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:47925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:50:59.450075 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37003
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:50:59.802331 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:50:59.802630 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:50:59.802731 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:50:59.810313 19872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:50:59.810443 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:50:59.810559 19872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:50:59.820637 19872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:50:59.821990 19872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:50:59.823235 19872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:50:59.836443 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:59.836380 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:50:59.836467 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:50:59.837256 19872 server_base.cc:1034] running on GCE node
I20260924 12:50:59.837930 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:50:59.838951 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:50:59.840210 19872 hybrid_clock.cc:648] HybridClock initialized: now 1790254259840151 us; error 51 us; skew 500 ppm
I20260924 12:50:59.840629 19872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:50:59.842936 19732 heartbeater.cc:503] Master 127.2.81.126:37003 was elected leader, sending a full tablet report...
Sep 24 12:50:59 dist-test-slave-9tm1 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:50:59.844558 19872 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:50:59.846421 19872 webserver.cc:460] Webserver started at http://127.2.81.67:43657/ using document root <none> and password file <none>
I20260924 12:50:59.847259 19872 fs_manager.cc:359] Metadata directory not provided
I20260924 12:50:59.847390 19872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:50:59.847675 19872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:50:59.851547 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "e6d1dc27d64b4e66ba5293c299373fea"
format_stamp: "Formatted at 2026-09-24 12:50:59 on dist-test-slave-9tm1"
server_key: "9cbed852002a9882d87531e7bc1cdee3"
server_key_iv: "71f46a599997f5eda8659facc2346ace"
server_key_version: "encryptionkey@0"
I20260924 12:50:59.852442 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "e6d1dc27d64b4e66ba5293c299373fea"
format_stamp: "Formatted at 2026-09-24 12:50:59 on dist-test-slave-9tm1"
server_key: "9cbed852002a9882d87531e7bc1cdee3"
server_key_iv: "71f46a599997f5eda8659facc2346ace"
server_key_version: "encryptionkey@0"
I20260924 12:50:59.858373 19872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 12:50:59.862293 19887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:59.863849 19872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:50:59.864043 19872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "e6d1dc27d64b4e66ba5293c299373fea"
format_stamp: "Formatted at 2026-09-24 12:50:59 on dist-test-slave-9tm1"
server_key: "9cbed852002a9882d87531e7bc1cdee3"
server_key_iv: "71f46a599997f5eda8659facc2346ace"
server_key_version: "encryptionkey@0"
I20260924 12:50:59.864259 19872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:50:59.885680 19872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:50:59.889978 19872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:50:59.890259 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:50:59.891258 19872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:50:59.892656 19872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:50:59.892733 19872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:59.892833 19872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:50:59.892895 19872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:50:59.923192 19872 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:36683
I20260924 12:50:59.923240 19999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:36683 every 8 connection(s)
I20260924 12:50:59.924520 19872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:50:59.926620 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 19872
I20260924 12:50:59.926812 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:50:59.927356 2373 external_mini_cluster.cc:1428] Setting key b694f2782a00b2a8f25f1bcd9636f4c9
Sep 24 12:50:59 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:50:59.947355 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:59.927694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:45261 (local address 127.2.81.126:37003)
0924 12:50:59.928171 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:59.928177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:59.929387 (+ 1210us) server_negotiation.cc:407] Connection header received
0924 12:50:59.931581 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:59.931586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:59.931683 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:59.931780 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:59.933599 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.934382 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.936001 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.936409 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.936540 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:59.940660 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:59.940666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:59.940669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:59.940711 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:59.944063 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:59.944855 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:59.944860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:59.944862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:59.945034 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:59.945421 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:59.945424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:59.945426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:59.945719 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:59.945838 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:59.946866 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0924 12:50:59.947077 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":439,"threads_started":1}
I20260924 12:50:59.948472 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:59.928326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37003 (local address 127.2.81.67:45261)
0924 12:50:59.929235 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0924 12:50:59.929284 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:50:59.931301 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:50:59.932064 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:50:59.932073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:50:59.932630 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:50:59.933409 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:59.933431 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.934552 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.934556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:59.935842 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:50:59.935856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.936664 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.936671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:50:59.936913 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:59.937717 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:50:59.937750 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:50:59.940320 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:50:59.944215 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:59.944223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:59.944239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:59.944733 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:59.945120 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:50:59.945124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:50:59.945126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:50:59.945323 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:50:59.945858 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:50:59.945865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:50:59.946335 (+ 470us) client_negotiation.cc:769] Sending connection context
0924 12:50:59.946719 (+ 384us) client_negotiation.cc:240] Negotiation successful
0924 12:50:59.947178 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":346,"threads_started":1}
I20260924 12:50:59.950465 20000 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37003
I20260924 12:50:59.950824 20000 heartbeater.cc:464] Registering TS with master...
I20260924 12:50:59.951747 20000 heartbeater.cc:511] Master 127.2.81.126:37003 requested a full tablet report, sending...
I20260924 12:50:59.954183 19543 ts_manager.cc:194] Registered new tserver with Master: e6d1dc27d64b4e66ba5293c299373fea (127.2.81.67:36683)
I20260924 12:50:59.955505 19543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:45261
I20260924 12:50:59.957435 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:50:59.975958 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:50:59.962686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.2.81.126:37003)
0924 12:50:59.962930 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 12:50:59.962937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:50:59.963262 (+ 325us) server_negotiation.cc:407] Connection header received
0924 12:50:59.963438 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:50:59.963442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:50:59.963550 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:50:59.963638 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:50:59.965154 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.965882 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.967210 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:50:59.967662 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:50:59.967778 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:50:59.970343 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:50:59.970349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:50:59.970352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:50:59.970392 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:50:59.973284 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:59.974008 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:59.974013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:59.974016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:59.974141 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:50:59.974504 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:50:59.974509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:50:59.974511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:50:59.974848 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:50:59.975083 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 12:50:59.975414 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 12:50:59.975651 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 12:50:59.980608 19543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:50:59.987941 19543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:51:00.029883 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.001574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:45591 (local address 127.0.0.1:38926)
0924 12:51:00.014092 (+ 12518us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:00.014158 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:00.014431 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:00.021981 (+ 7550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:00.021986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:00.022019 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:00.022419 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.022429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.023974 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.023980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.026879 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.026916 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.028527 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.028534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.028801 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.028863 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 12:51:00.029082 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 12:51:00.029307 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11955,"mutex_wait_us":165,"thread_start_us":18214,"threads_started":1}
I20260924 12:51:00.030189 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.002897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.2.81.66:35975)
0924 12:51:00.018278 (+ 15381us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.018287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.020477 (+ 2190us) server_negotiation.cc:407] Connection header received
0924 12:51:00.020639 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.020646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.020872 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.021139 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:00.021880 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.023398 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.027187 (+ 3789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.028721 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.028870 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.029089 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.029379 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.029565 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15236,"thread_start_us":15105,"threads_started":1}
I20260924 12:51:00.029904 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.002305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35975 (local address 127.0.0.1:60588)
0924 12:51:00.020214 (+ 17909us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:00.020233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:00.020355 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:00.021147 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:00.021154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:00.021185 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:00.021606 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.021616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.023570 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.023575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.026939 (+ 3364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.026949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.029011 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.029020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.029180 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.029214 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:51:00.029292 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 12:51:00.029413 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17649,"mutex_wait_us":190,"thread_start_us":16588,"threads_started":1}
I20260924 12:51:00.030946 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.001983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.2.81.65:45591)
0924 12:51:00.018762 (+ 16779us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.018771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.018804 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:51:00.018929 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.018938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.021145 (+ 2207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.021349 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:00.022570 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.023747 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.027203 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.028297 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.028438 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.028713 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.030396 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.030602 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16618,"thread_start_us":16493,"threads_started":1}
I20260924 12:51:00.033263 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.001873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:36683 (local address 127.0.0.1:51944)
0924 12:51:00.019235 (+ 17362us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:00.019361 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:00.019564 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:00.021231 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:00.021236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:00.021274 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:00.022148 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.022165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.024881 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.024887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.026876 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.026899 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.031826 (+ 4927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.031836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.032047 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.032086 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 12:51:00.032196 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 12:51:00.032531 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17155,"mutex_wait_us":186,"thread_start_us":11397,"threads_started":1}
I20260924 12:51:00.035176 19801 tablet_service.cc:1467] Processing CreateTablet for tablet 510eace7f060431796a38801ed290fc5 (DEFAULT_TABLE table=test-table [id=970ce04c23144fea99d8d38b29629c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:00.035168 19667 tablet_service.cc:1467] Processing CreateTablet for tablet 510eace7f060431796a38801ed290fc5 (DEFAULT_TABLE table=test-table [id=970ce04c23144fea99d8d38b29629c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:00.036691 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510eace7f060431796a38801ed290fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:00.037094 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510eace7f060431796a38801ed290fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:00.037537 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.003365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.2.81.67:36683)
0924 12:51:00.017853 (+ 14488us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.017864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.019521 (+ 1657us) server_negotiation.cc:407] Connection header received
0924 12:51:00.019813 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.019821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.020310 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.022381 (+ 2071us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:00.022693 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.024735 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.027178 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.031442 (+ 4264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.031639 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.031958 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.036735 (+ 4777us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.037156 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14256,"thread_start_us":14033,"threads_started":1}
I20260924 12:51:00.042831 19935 tablet_service.cc:1467] Processing CreateTablet for tablet 510eace7f060431796a38801ed290fc5 (DEFAULT_TABLE table=test-table [id=970ce04c23144fea99d8d38b29629c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:00.044636 19935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 510eace7f060431796a38801ed290fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:00.050716 20019 tablet_bootstrap.cc:492] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Bootstrap starting.
I20260924 12:51:00.054482 20020 tablet_bootstrap.cc:492] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Bootstrap starting.
I20260924 12:51:00.056300 20019 tablet_bootstrap.cc:654] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:00.057898 20019 log.cc:824] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:00.058761 20020 tablet_bootstrap.cc:654] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:00.060575 20020 log.cc:824] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:00.061101 20022 tablet_bootstrap.cc:492] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Bootstrap starting.
I20260924 12:51:00.064543 20019 tablet_bootstrap.cc:492] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: No bootstrap required, opened a new log
I20260924 12:51:00.064785 20022 tablet_bootstrap.cc:654] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:00.064824 20019 ts_tablet_manager.cc:1397] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260924 12:51:00.064893 20020 tablet_bootstrap.cc:492] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: No bootstrap required, opened a new log
I20260924 12:51:00.065619 20020 ts_tablet_manager.cc:1397] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260924 12:51:00.066907 20022 log.cc:824] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:00.071471 20022 tablet_bootstrap.cc:492] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: No bootstrap required, opened a new log
I20260924 12:51:00.071834 20022 ts_tablet_manager.cc:1397] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 12:51:00.074404 20020 raft_consensus.cc:348] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.074813 20020 raft_consensus.cc:374] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:00.074905 20020 raft_consensus.cc:729] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebf579d532743628c9d7ec8a48fdc7a, State: Initialized, Role: FOLLOWER
I20260924 12:51:00.075510 20020 consensus_queue.cc:260] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.076040 20019 raft_consensus.cc:348] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.076514 20019 raft_consensus.cc:374] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:00.076614 20019 raft_consensus.cc:729] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd5fd482fbe43579a6db44a329f1a68, State: Initialized, Role: FOLLOWER
I20260924 12:51:00.077309 20020 ts_tablet_manager.cc:1428] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 12:51:00.077387 20019 consensus_queue.cc:260] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.078589 19866 heartbeater.cc:503] Master 127.2.81.126:37003 was elected leader, sending a full tablet report...
I20260924 12:51:00.079246 20019 ts_tablet_manager.cc:1428] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 12:51:00.082410 20022 raft_consensus.cc:348] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.082990 20022 raft_consensus.cc:374] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:00.083098 20022 raft_consensus.cc:729] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d1dc27d64b4e66ba5293c299373fea, State: Initialized, Role: FOLLOWER
I20260924 12:51:00.083797 20022 consensus_queue.cc:260] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.085021 20000 heartbeater.cc:503] Master 127.2.81.126:37003 was elected leader, sending a full tablet report...
I20260924 12:51:00.085515 20022 ts_tablet_manager.cc:1428] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260924 12:51:00.133185 19867 tablet.cc:2369] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:51:00.176234 20001 tablet.cc:2369] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:00.204348 20025 raft_consensus.cc:482] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:51:00.204638 20025 raft_consensus.cc:504] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.206352 20025 leader_election.cc:288] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6d1dc27d64b4e66ba5293c299373fea (127.2.81.67:36683), 1dd5fd482fbe43579a6db44a329f1a68 (127.2.81.66:35975)
I20260924 12:51:00.213467 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.206821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:38271 (local address 127.2.81.66:35975)
0924 12:51:00.207018 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.207024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.207578 (+ 554us) server_negotiation.cc:407] Connection header received
0924 12:51:00.207758 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.207764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.207847 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.207986 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:00.208643 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.209505 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.211604 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.212432 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.212538 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.212619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.213043 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.213184 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 12:51:00.213472 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.206843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:43985 (local address 127.2.81.67:36683)
0924 12:51:00.207186 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.207192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.208012 (+ 820us) server_negotiation.cc:407] Connection header received
0924 12:51:00.208093 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.208098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.208174 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.208280 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:00.208887 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.209842 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.211754 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.212569 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.212697 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.212798 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.213089 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.213216 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260924 12:51:00.213639 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.206668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:36683 (local address 127.2.81.65:43985)
0924 12:51:00.207825 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:00.207842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:00.207989 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:00.208293 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:00.208296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:00.208319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:00.208737 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.208749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.209991 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.209996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.211602 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.211615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.212785 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.212789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.212918 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.212943 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:51:00.213051 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 12:51:00.213184 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":39,"thread_start_us":361,"threads_started":1}
I20260924 12:51:00.214071 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.206668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35975 (local address 127.2.81.65:38271)
0924 12:51:00.207413 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:00.207431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:00.207617 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:00.208008 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:00.208013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:00.208041 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:00.208478 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.208493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.209644 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.209650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.211439 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:00.211451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.212659 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.212665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:00.212853 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.212884 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:51:00.212962 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 12:51:00.213127 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":87,"thread_start_us":506,"threads_started":1}
I20260924 12:51:00.214650 19955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510eace7f060431796a38801ed290fc5" candidate_uuid: "aebf579d532743628c9d7ec8a48fdc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d1dc27d64b4e66ba5293c299373fea" is_pre_election: true
I20260924 12:51:00.215024 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510eace7f060431796a38801ed290fc5" candidate_uuid: "aebf579d532743628c9d7ec8a48fdc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd5fd482fbe43579a6db44a329f1a68" is_pre_election: true
I20260924 12:51:00.215147 19955 raft_consensus.cc:2454] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aebf579d532743628c9d7ec8a48fdc7a in term 0.
I20260924 12:51:00.215468 19821 raft_consensus.cc:2454] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aebf579d532743628c9d7ec8a48fdc7a in term 0.
I20260924 12:51:00.215778 19623 leader_election.cc:302] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aebf579d532743628c9d7ec8a48fdc7a, e6d1dc27d64b4e66ba5293c299373fea; no voters:
I20260924 12:51:00.216166 20025 raft_consensus.cc:2781] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:51:00.216297 20025 raft_consensus.cc:482] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:51:00.216396 20025 raft_consensus.cc:3037] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:00.218701 20025 raft_consensus.cc:504] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.219352 20025 leader_election.cc:288] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [CANDIDATE]: Term 1 election: Requested vote from peers e6d1dc27d64b4e66ba5293c299373fea (127.2.81.67:36683), 1dd5fd482fbe43579a6db44a329f1a68 (127.2.81.66:35975)
I20260924 12:51:00.219924 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510eace7f060431796a38801ed290fc5" candidate_uuid: "aebf579d532743628c9d7ec8a48fdc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd5fd482fbe43579a6db44a329f1a68"
I20260924 12:51:00.219959 19955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "510eace7f060431796a38801ed290fc5" candidate_uuid: "aebf579d532743628c9d7ec8a48fdc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d1dc27d64b4e66ba5293c299373fea"
I20260924 12:51:00.220134 19821 raft_consensus.cc:3037] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:00.220170 19955 raft_consensus.cc:3037] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:00.222512 19821 raft_consensus.cc:2454] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aebf579d532743628c9d7ec8a48fdc7a in term 1.
I20260924 12:51:00.222537 19955 raft_consensus.cc:2454] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aebf579d532743628c9d7ec8a48fdc7a in term 1.
I20260924 12:51:00.223042 19620 leader_election.cc:302] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd5fd482fbe43579a6db44a329f1a68, aebf579d532743628c9d7ec8a48fdc7a; no voters:
I20260924 12:51:00.223366 20025 raft_consensus.cc:2781] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:00.223845 20025 raft_consensus.cc:686] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 1 LEADER]: Becoming Leader. State: Replica: aebf579d532743628c9d7ec8a48fdc7a, State: Running, Role: LEADER
I20260924 12:51:00.224408 20025 consensus_queue.cc:237] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } }
I20260924 12:51:00.229035 19541 catalog_manager.cc:5495] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a reported cstate change: term changed from 0 to 1, leader changed from <none> to aebf579d532743628c9d7ec8a48fdc7a (127.2.81.65). New cstate: current_term: 1 leader_uuid: "aebf579d532743628c9d7ec8a48fdc7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aebf579d532743628c9d7ec8a48fdc7a" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 45591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 } health_report { overall_health: UNKNOWN } } }
I20260924 12:51:00.270200 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.263772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.2.81.65:45591)
0924 12:51:00.263966 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.263971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.263989 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:51:00.264197 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.264202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.264285 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.264427 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:00.264911 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.265757 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.267238 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.267699 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.267813 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.268450 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:00.269249 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:00.269382 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.269624 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.269716 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 12:51:00.279104 19955 raft_consensus.cc:1261] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 1 FOLLOWER]: Refusing update from remote peer aebf579d532743628c9d7ec8a48fdc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:00.279188 19821 raft_consensus.cc:1261] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 1 FOLLOWER]: Refusing update from remote peer aebf579d532743628c9d7ec8a48fdc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:00.280145 20030 consensus_queue.cc:1035] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d1dc27d64b4e66ba5293c299373fea" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:00.280543 20025 consensus_queue.cc:1035] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd5fd482fbe43579a6db44a329f1a68" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:00.292691 20033 mvcc.cc:205] Tried to move back new op lower bound from 7332881450092118016 to 7332881449883414528. Current Snapshot: MvccSnapshot[applied={T|T < 7332881450092118016}]
I20260924 12:51:00.307363 20034 mvcc.cc:205] Tried to move back new op lower bound from 7332881450092118016 to 7332881449883414528. Current Snapshot: MvccSnapshot[applied={T|T < 7332881450092118016}]
I20260924 12:51:00.311354 19542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:00.311704 19542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:00.315690 19542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 737d82cbc30b4a15937a57b99c8f360e: Sending DeleteTablet for 3 replicas of tablet 510eace7f060431796a38801ed290fc5
I20260924 12:51:00.316960 19935 tablet_service.cc:1514] Processing DeleteTablet for tablet 510eace7f060431796a38801ed290fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:00 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:51944
I20260924 12:51:00.317001 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet 510eace7f060431796a38801ed290fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:00 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:38926
I20260924 12:51:00.317067 19801 tablet_service.cc:1514] Processing DeleteTablet for tablet 510eace7f060431796a38801ed290fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:00 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:60588
I20260924 12:51:00.319657 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19604
I20260924 12:51:00.327791 20045 tablet_replica.cc:330] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: stopping tablet replica
I20260924 12:51:00.328286 20045 raft_consensus.cc:2229] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:00.328850 20045 raft_consensus.cc:2258] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:00.329766 20044 tablet_replica.cc:330] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: stopping tablet replica
I20260924 12:51:00.330196 20044 raft_consensus.cc:2229] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:51:00.330892 20044 raft_consensus.cc:2258] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:00.333339 20045 ts_tablet_manager.cc:1905] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:00.339221 20044 ts_tablet_manager.cc:1905] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:00.342816 20043 tablet_replica.cc:330] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: stopping tablet replica
I20260924 12:51:00.343319 20043 raft_consensus.cc:2229] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:00.343935 20043 raft_consensus.cc:2258] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:00.346411 20043 ts_tablet_manager.cc:1905] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:00.346777 20045 ts_tablet_manager.cc:1918] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:00.346952 20045 log.cc:1196] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/510eace7f060431796a38801ed290fc5
I20260924 12:51:00.347468 20045 ts_tablet_manager.cc:1939] T 510eace7f060431796a38801ed290fc5 P 1dd5fd482fbe43579a6db44a329f1a68: Deleting consensus metadata
Sep 24 12:51:00 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:51:00.349678 19527 catalog_manager.cc:4841] TS 1dd5fd482fbe43579a6db44a329f1a68 (127.2.81.66:35975): tablet 510eace7f060431796a38801ed290fc5 (table test-table [id=970ce04c23144fea99d8d38b29629c24]) successfully deleted
I20260924 12:51:00.353160 20044 ts_tablet_manager.cc:1918] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:00.353344 20044 log.cc:1196] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/510eace7f060431796a38801ed290fc5
I20260924 12:51:00.354269 20044 ts_tablet_manager.cc:1939] T 510eace7f060431796a38801ed290fc5 P aebf579d532743628c9d7ec8a48fdc7a: Deleting consensus metadata
I20260924 12:51:00.356613 19529 catalog_manager.cc:4841] TS aebf579d532743628c9d7ec8a48fdc7a (127.2.81.65:45591): tablet 510eace7f060431796a38801ed290fc5 (table test-table [id=970ce04c23144fea99d8d38b29629c24]) successfully deleted
I20260924 12:51:00.356686 20043 ts_tablet_manager.cc:1918] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:00.356874 20043 log.cc:1196] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/510eace7f060431796a38801ed290fc5
I20260924 12:51:00.357690 20043 ts_tablet_manager.cc:1939] T 510eace7f060431796a38801ed290fc5 P e6d1dc27d64b4e66ba5293c299373fea: Deleting consensus metadata
I20260924 12:51:00.359302 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.321240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.2.81.65:45591)
0924 12:51:00.321437 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.321444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.335583 (+ 14139us) server_negotiation.cc:407] Connection header received
0924 12:51:00.336112 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.336119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.336230 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.336308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:00.338300 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.339539 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.342096 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.342812 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.342940 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.349586 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:00.349601 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:00.349618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:00.349681 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:00.354727 (+ 5046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:00.355820 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:00.355832 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:00.355835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:00.356020 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:00.356631 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:00.356636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:00.356639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:00.357547 (+ 908us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:00.357966 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.358641 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.358925 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:51:00.359742 19530 catalog_manager.cc:4841] TS e6d1dc27d64b4e66ba5293c299373fea (127.2.81.67:36683): tablet 510eace7f060431796a38801ed290fc5 (table test-table [id=970ce04c23144fea99d8d38b29629c24]) successfully deleted
I20260924 12:51:00.360360 19727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:00.462780 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19604
I20260924 12:51:00.479511 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19738
Sep 24 12:51:00 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:51:00.497437 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.480399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.2.81.66:35975)
0924 12:51:00.480630 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.480636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.480657 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:51:00.480739 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.480748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.480841 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.480925 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:00.482523 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.483272 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.484617 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.485066 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.485177 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.490732 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:00.490758 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:00.490781 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:00.490956 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:00.494485 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:00.495311 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:00.495319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:00.495321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:00.495493 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:00.495929 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:00.495934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:00.495936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:00.496352 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:00.496643 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.496921 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.497176 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 12:51:00.498266 19861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:00.589242 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19738
I20260924 12:51:00.606172 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19872
Sep 24 12:51:00 dist-test-slave-9tm1 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:51:00.623764 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:00.607023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.2.81.67:36683)
0924 12:51:00.607205 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:00.607210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:00.607228 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:51:00.607350 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:00.607354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:00.607421 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:00.607515 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:00.608965 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.609732 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.611126 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:00.611693 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:00.611818 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:00.617173 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:00.617192 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:00.617210 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:00.617252 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:00.620250 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:00.621349 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:00.621358 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:00.621360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:00.621613 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:00.622083 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:00.622087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:00.622090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:00.622464 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:00.622776 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:00.623124 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 12:51:00.623315 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:51:00.624437 19995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:00.715615 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19872
I20260924 12:51:00.728408 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 19511
I20260924 12:51:00.729557 19573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:00.842697 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 19511
2026-09-24T12:51:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4508 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T12:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:00Z Disabled control of system clock
I20260924 12:51:00.916503 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45723
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:45723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 12:51:01.261726 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:01.262035 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:01.262120 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:01.269603 20054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:01.269728 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:01.269764 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:01.269822 20054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:01.269866 20054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:01.279600 20054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:45723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:45723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:01.280876 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:01.282747 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:01.294677 20059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:01.294863 20062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:01.294909 20060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:01.295728 20054 server_base.cc:1034] running on GCE node
I20260924 12:51:01.296552 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:01.298081 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:01.299295 20054 hybrid_clock.cc:648] HybridClock initialized: now 1790254261299235 us; error 60 us; skew 500 ppm
I20260924 12:51:01.299687 20054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:01.302322 20054 webserver.cc:460] Webserver started at http://127.2.81.126:36747/ using document root <none> and password file <none>
I20260924 12:51:01.303124 20054 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:01.303226 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:01.303524 20054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:01.306756 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "b11c81e35e8643d2ae81243409013480"
format_stamp: "Formatted at 2026-09-24 12:51:01 on dist-test-slave-9tm1"
server_key: "4a8c0343fa24fb0113273bad62f0a08e"
server_key_iv: "a45e99853ee01ac522e7141eb1308b10"
server_key_version: "encryptionkey@0"
I20260924 12:51:01.307605 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "b11c81e35e8643d2ae81243409013480"
format_stamp: "Formatted at 2026-09-24 12:51:01 on dist-test-slave-9tm1"
server_key: "4a8c0343fa24fb0113273bad62f0a08e"
server_key_iv: "a45e99853ee01ac522e7141eb1308b10"
server_key_version: "encryptionkey@0"
I20260924 12:51:01.313555 20054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 12:51:01.317513 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:01.319139 20054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:51:01.319312 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "b11c81e35e8643d2ae81243409013480"
format_stamp: "Formatted at 2026-09-24 12:51:01 on dist-test-slave-9tm1"
server_key: "4a8c0343fa24fb0113273bad62f0a08e"
server_key_iv: "a45e99853ee01ac522e7141eb1308b10"
server_key_version: "encryptionkey@0"
I20260924 12:51:01.319526 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:01.339556 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:01.340396 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:01.340651 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:01.359040 20054 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:45723
I20260924 12:51:01.359184 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:45723 every 8 connection(s)
I20260924 12:51:01.360435 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:01.365854 20120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:01.370359 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20054
I20260924 12:51:01.370561 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:01.371058 2373 external_mini_cluster.cc:1428] Setting key 60a62969d00ed12b390d118748da8aa4
I20260924 12:51:01.377622 20120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480: Bootstrap starting.
I20260924 12:51:01.382011 20120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:01.383399 20120 log.cc:824] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:01.386255 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:01.373051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.2.81.126:45723)
0924 12:51:01.374234 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:01.374245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:01.374293 (+ 48us) server_negotiation.cc:407] Connection header received
0924 12:51:01.376180 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:01.376206 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:01.376840 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:01.377341 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:01.377949 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:01.379871 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:01.381348 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:01.382096 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:01.382282 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:01.383609 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:01.383624 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:01.383643 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:01.384363 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:01.384468 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:01.384512 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:01.384652 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:01.384915 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 12:51:01.385292 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":427,"threads_started":1}
I20260924 12:51:01.386817 20120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480: No bootstrap required, opened a new log
I20260924 12:51:01.395887 20120 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11c81e35e8643d2ae81243409013480" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45723 } }
I20260924 12:51:01.396327 20120 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:01.396446 20120 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11c81e35e8643d2ae81243409013480, State: Initialized, Role: FOLLOWER
I20260924 12:51:01.397279 20120 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11c81e35e8643d2ae81243409013480" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45723 } }
I20260924 12:51:01.397552 20120 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:01.397701 20120 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:01.397866 20120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:01.400403 20120 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11c81e35e8643d2ae81243409013480" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45723 } }
I20260924 12:51:01.401081 20120 leader_election.cc:302] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11c81e35e8643d2ae81243409013480; no voters:
I20260924 12:51:01.401747 20120 leader_election.cc:288] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:01.401927 20125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:01.402748 20125 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [term 1 LEADER]: Becoming Leader. State: Replica: b11c81e35e8643d2ae81243409013480, State: Running, Role: LEADER
I20260924 12:51:01.403280 20125 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11c81e35e8643d2ae81243409013480" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45723 } }
I20260924 12:51:01.403911 20120 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:01.406482 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11c81e35e8643d2ae81243409013480. Latest consensus state: current_term: 1 leader_uuid: "b11c81e35e8643d2ae81243409013480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11c81e35e8643d2ae81243409013480" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45723 } } }
I20260924 12:51:01.407202 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:01.406664 20126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11c81e35e8643d2ae81243409013480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11c81e35e8643d2ae81243409013480" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 45723 } } }
I20260924 12:51:01.407752 20126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:01.409075 20134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:01.414104 20134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:01.424009 20134 catalog_manager.cc:1347] Generated new cluster ID: 3c8f9e3eb63741c6822a953a9eaa7f75
I20260924 12:51:01.424158 20134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:01.457199 20134 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:01.458534 20134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:01.468246 20134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480: Generated new TSK 0
I20260924 12:51:01.469230 20134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:01.481839 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45723
--builtin_ntp_servers=127.2.81.84:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 12:51:01.850050 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:01.850375 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:01.850498 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:01.859331 20144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:01.859484 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:01.859644 20144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:51:01.870299 20144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:01.871644 20144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:01.873463 20144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:01.887408 20150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:01.887593 20152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:01.888058 20149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:01.889153 20144 server_base.cc:1034] running on GCE node
I20260924 12:51:01.889940 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:01.890980 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:01.892225 20144 hybrid_clock.cc:648] HybridClock initialized: now 1790254261892161 us; error 58 us; skew 500 ppm
I20260924 12:51:01.892627 20144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:01.895118 20144 webserver.cc:460] Webserver started at http://127.2.81.65:37623/ using document root <none> and password file <none>
I20260924 12:51:01.895826 20144 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:01.895933 20144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:01.896242 20144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:01.899562 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "96e10d1be30949cca143d724c4e6901d"
format_stamp: "Formatted at 2026-09-24 12:51:01 on dist-test-slave-9tm1"
server_key: "73e6e0b84f25a1479943281b31e73f17"
server_key_iv: "ff43783285adbcdc9c6edd4fb7cf7389"
server_key_version: "encryptionkey@0"
I20260924 12:51:01.900483 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "96e10d1be30949cca143d724c4e6901d"
format_stamp: "Formatted at 2026-09-24 12:51:01 on dist-test-slave-9tm1"
server_key: "73e6e0b84f25a1479943281b31e73f17"
server_key_iv: "ff43783285adbcdc9c6edd4fb7cf7389"
server_key_version: "encryptionkey@0"
I20260924 12:51:01.906913 20144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260924 12:51:01.910895 20158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:01.912406 20144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:01.912588 20144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "96e10d1be30949cca143d724c4e6901d"
format_stamp: "Formatted at 2026-09-24 12:51:01 on dist-test-slave-9tm1"
server_key: "73e6e0b84f25a1479943281b31e73f17"
server_key_iv: "ff43783285adbcdc9c6edd4fb7cf7389"
server_key_version: "encryptionkey@0"
I20260924 12:51:01.912812 20144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:01.933920 20144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:01.934731 20144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:01.934978 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:01.935982 20144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:01.937517 20144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:01.937608 20144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:01.937683 20144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:01.937734 20144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:01.970379 20144 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:40593
I20260924 12:51:01.970515 20270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:40593 every 8 connection(s)
I20260924 12:51:01.971717 20144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:51:01.980484 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20144
I20260924 12:51:01.980671 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:51:01.981132 2373 external_mini_cluster.cc:1428] Setting key 59ccca92650f8b6db36902311bcd153d
I20260924 12:51:01.984503 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45723
--builtin_ntp_servers=127.2.81.84:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 12:51:01.986351 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:01.974697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:60209 (local address 127.2.81.126:45723)
0924 12:51:01.975168 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:01.975175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:01.976476 (+ 1301us) server_negotiation.cc:407] Connection header received
0924 12:51:01.978453 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:01.978458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:01.978538 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:01.978653 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:01.980069 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:01.981075 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:01.983116 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:01.983652 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:01.983764 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:01.985541 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:01.985546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:01.985550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:01.985569 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:01.985620 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:01.985633 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:01.985712 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:01.985982 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 12:51:01.986079 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260924 12:51:01.987738 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:01.975284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45723 (local address 127.2.81.65:60209)
0924 12:51:01.976275 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:01.976314 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:01.978122 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:01.978958 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:01.978968 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:01.979073 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:01.979729 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:01.979750 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:01.981279 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:01.981285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:01.982928 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:01.982954 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:01.983906 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:01.983914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:01.984256 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:01.985103 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:01.985108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:01.985167 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:01.985287 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:01.985731 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:01.985741 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:01.985750 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:51:01.985948 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 12:51:01.986396 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":320,"threads_started":1}
I20260924 12:51:01.989729 20271 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45723
I20260924 12:51:01.990090 20271 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:01.990926 20271 heartbeater.cc:511] Master 127.2.81.126:45723 requested a full tablet report, sending...
I20260924 12:51:01.993038 20085 ts_manager.cc:194] Registered new tserver with Master: 96e10d1be30949cca143d724c4e6901d (127.2.81.65:40593)
I20260924 12:51:01.994536 20085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.65:60209
W20260924 12:51:02.328040 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:02.328341 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:02.328462 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:02.335801 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:02.335922 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:02.336058 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:51:02.345619 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:02.346863 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:02.348544 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:02.361497 20281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:02.361467 20280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:02.362246 20283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:02.362386 20275 server_base.cc:1034] running on GCE node
I20260924 12:51:02.363037 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:02.364039 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:02.365274 20275 hybrid_clock.cc:648] HybridClock initialized: now 1790254262365217 us; error 45 us; skew 500 ppm
I20260924 12:51:02.365646 20275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:02.368130 20275 webserver.cc:460] Webserver started at http://127.2.81.66:44891/ using document root <none> and password file <none>
I20260924 12:51:02.368858 20275 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:02.368950 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:02.369277 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:02.372159 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "d74cb8af4f5c405eac0e0ca1f083730b"
format_stamp: "Formatted at 2026-09-24 12:51:02 on dist-test-slave-9tm1"
server_key: "c9592d39c30590375c9e6d9e0e82ed4b"
server_key_iv: "52c0f4dab31dbf2bbf9a5ac02612e765"
server_key_version: "encryptionkey@0"
I20260924 12:51:02.372959 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "d74cb8af4f5c405eac0e0ca1f083730b"
format_stamp: "Formatted at 2026-09-24 12:51:02 on dist-test-slave-9tm1"
server_key: "c9592d39c30590375c9e6d9e0e82ed4b"
server_key_iv: "52c0f4dab31dbf2bbf9a5ac02612e765"
server_key_version: "encryptionkey@0"
I20260924 12:51:02.378072 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 12:51:02.381639 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:02.382951 20275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:02.383096 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "d74cb8af4f5c405eac0e0ca1f083730b"
format_stamp: "Formatted at 2026-09-24 12:51:02 on dist-test-slave-9tm1"
server_key: "c9592d39c30590375c9e6d9e0e82ed4b"
server_key_iv: "52c0f4dab31dbf2bbf9a5ac02612e765"
server_key_version: "encryptionkey@0"
I20260924 12:51:02.383288 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:02.394806 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:02.395661 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:02.395924 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:02.396895 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:02.398448 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:02.398547 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:02.398655 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:02.398729 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:02.428435 20275 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:40787
I20260924 12:51:02.428479 20401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:40787 every 8 connection(s)
I20260924 12:51:02.429746 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:51:02.439987 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20275
I20260924 12:51:02.440140 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:51:02.440663 2373 external_mini_cluster.cc:1428] Setting key e3730713e92fba1d76b447b424a8c761
I20260924 12:51:02.441819 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.432182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:47921 (local address 127.2.81.126:45723)
0924 12:51:02.432353 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:02.432357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:02.433744 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 12:51:02.435435 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:02.435440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:02.435512 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:02.435602 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:02.436670 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.437431 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.438979 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.439382 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.439477 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.440923 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:02.440930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:02.440934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:02.440958 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:02.441038 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:02.441057 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:02.441148 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:02.441386 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 12:51:02.441471 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:51:02.443053 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.432627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45723 (local address 127.2.81.66:47921)
0924 12:51:02.433585 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:02.433620 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:02.435197 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:02.435850 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:02.435863 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:02.435964 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:02.436535 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.436552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.437579 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.437583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.438860 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.438873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.439692 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.439697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.439908 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.440594 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:02.440597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:02.440639 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:02.440719 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:02.441161 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:02.441169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:02.441176 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:51:02.441384 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 12:51:02.441762 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":341,"threads_started":1}
I20260924 12:51:02.444077 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45723
--builtin_ntp_servers=127.2.81.84:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 12:51:02.444778 20402 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45723
I20260924 12:51:02.445148 20402 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:02.446076 20402 heartbeater.cc:511] Master 127.2.81.126:45723 requested a full tablet report, sending...
I20260924 12:51:02.447772 20085 ts_manager.cc:194] Registered new tserver with Master: d74cb8af4f5c405eac0e0ca1f083730b (127.2.81.66:40787)
I20260924 12:51:02.448640 20085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.66:47921
W20260924 12:51:02.778403 20405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:02.778883 20405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:02.779036 20405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:02.786763 20405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:02.786877 20405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:02.786994 20405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:51:02.796792 20405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:45723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:02.798111 20405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:02.799998 20405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:02.813930 20413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:02.813946 20410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:02.814239 20411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:02.814759 20405 server_base.cc:1034] running on GCE node
I20260924 12:51:02.815399 20405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:02.816468 20405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:02.817708 20405 hybrid_clock.cc:648] HybridClock initialized: now 1790254262817652 us; error 49 us; skew 500 ppm
I20260924 12:51:02.818029 20405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:02.820459 20405 webserver.cc:460] Webserver started at http://127.2.81.67:45245/ using document root <none> and password file <none>
I20260924 12:51:02.821352 20405 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:02.821463 20405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:02.821784 20405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:02.824781 20405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "80d54cdbc5a640a8b691a48e088fd754"
format_stamp: "Formatted at 2026-09-24 12:51:02 on dist-test-slave-9tm1"
server_key: "e68720f90a98e4979179fccc1e406a02"
server_key_iv: "6a68ac4acfb95a69eee253bf350adba3"
server_key_version: "encryptionkey@0"
I20260924 12:51:02.825619 20405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "80d54cdbc5a640a8b691a48e088fd754"
format_stamp: "Formatted at 2026-09-24 12:51:02 on dist-test-slave-9tm1"
server_key: "e68720f90a98e4979179fccc1e406a02"
server_key_iv: "6a68ac4acfb95a69eee253bf350adba3"
server_key_version: "encryptionkey@0"
I20260924 12:51:02.830901 20405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:02.834898 20419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:02.836484 20405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:02.836688 20405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "80d54cdbc5a640a8b691a48e088fd754"
format_stamp: "Formatted at 2026-09-24 12:51:02 on dist-test-slave-9tm1"
server_key: "e68720f90a98e4979179fccc1e406a02"
server_key_iv: "6a68ac4acfb95a69eee253bf350adba3"
server_key_version: "encryptionkey@0"
I20260924 12:51:02.836917 20405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:02.854995 20405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:02.855794 20405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:02.856036 20405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:02.857029 20405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:02.858584 20405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:02.858681 20405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:02.858762 20405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:02.858814 20405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:02.891412 20405 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:41931
I20260924 12:51:02.891525 20531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:41931 every 8 connection(s)
I20260924 12:51:02.892735 20405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:51:02.898766 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20405
I20260924 12:51:02.898947 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:51:02.899420 2373 external_mini_cluster.cc:1428] Setting key ccad0ad320b2cebdbb53d6e6346a4028
I20260924 12:51:02.906054 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.895422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:56613 (local address 127.2.81.126:45723)
0924 12:51:02.895609 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:02.895613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:02.896921 (+ 1308us) server_negotiation.cc:407] Connection header received
0924 12:51:02.898772 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:02.898778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:02.898870 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:02.898987 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:02.900196 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.901321 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.903011 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.903415 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.903526 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.905272 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:02.905278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:02.905284 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:02.905307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:02.905340 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:02.905353 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:02.905430 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:02.905700 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 12:51:02.905793 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:51:02.907234 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.895869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:45723 (local address 127.2.81.67:56613)
0924 12:51:02.896770 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:02.896810 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:02.898479 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:02.899327 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:02.899335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:02.899440 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:02.900044 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.900062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.901463 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.901467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.902859 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.902874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.903640 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.903647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.903873 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.904688 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:02.904690 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:02.904746 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:02.904836 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:02.905472 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:02.905483 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:02.905490 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:51:02.905671 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 12:51:02.906130 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260924 12:51:02.909006 20532 heartbeater.cc:347] Connected to a master server at 127.2.81.126:45723
I20260924 12:51:02.909443 20532 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:02.910246 20532 heartbeater.cc:511] Master 127.2.81.126:45723 requested a full tablet report, sending...
I20260924 12:51:02.911864 20085 ts_manager.cc:194] Registered new tserver with Master: 80d54cdbc5a640a8b691a48e088fd754 (127.2.81.67:41931)
I20260924 12:51:02.912676 20085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.67:56613
I20260924 12:51:02.915671 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:51:02.926157 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.920008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.2.81.126:45723)
0924 12:51:02.920268 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:02.920275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:02.920549 (+ 274us) server_negotiation.cc:407] Connection header received
0924 12:51:02.920721 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:02.920725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:02.920801 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:02.920928 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:02.921525 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.922259 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.923811 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.924246 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.924369 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.925477 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:02.925481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:02.925485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:02.925504 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:02.925537 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:02.925548 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:02.925623 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:02.925749 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 12:51:02.925849 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 12:51:02.930975 20085 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:51:02.938243 20085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:51:02.963497 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.951498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:40787 (local address 127.0.0.1:45850)
0924 12:51:02.952491 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:02.952531 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:02.952733 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:02.955413 (+ 2680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:02.955418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:02.955448 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:02.955878 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.955892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.958168 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.958174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.960595 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.960613 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.962497 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.962504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.962712 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.962768 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 12:51:02.962893 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 12:51:02.963107 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":29,"thread_start_us":364,"threads_started":1}
I20260924 12:51:02.963490 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.952303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.2.81.66:40787)
0924 12:51:02.954853 (+ 2550us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:02.954869 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:02.954893 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:51:02.954986 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:02.954993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:02.955189 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:02.956066 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:02.956290 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.958013 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.960865 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.962261 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.962394 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.962704 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:02.962960 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 12:51:02.963139 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2419,"thread_start_us":2284,"threads_started":1}
I20260924 12:51:02.964915 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.952893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55940)
0924 12:51:02.953873 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:02.953903 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:02.954026 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:02.956607 (+ 2581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:02.956612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:02.956639 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:02.956960 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.956997 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.959385 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.959391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.961793 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.961808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.964090 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.964097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.964262 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.964297 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:51:02.964372 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 12:51:02.964523 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":1,"thread_start_us":1845,"threads_started":1}
I20260924 12:51:02.965732 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.951598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:41931 (local address 127.0.0.1:43004)
0924 12:51:02.953384 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:02.953414 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:02.953526 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:02.955230 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:02.955239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:02.955273 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:02.955731 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.955748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.958430 (+ 2682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.958436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.960815 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:02.960834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.964379 (+ 3545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.964385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:02.964549 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.964583 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:51:02.964654 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 12:51:02.965310 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"mutex_wait_us":35,"thread_start_us":911,"threads_started":1}
I20260924 12:51:02.965768 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.953362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.2.81.65:40593)
0924 12:51:02.956029 (+ 2667us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:02.956037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:02.956062 (+ 25us) server_negotiation.cc:407] Connection header received
0924 12:51:02.956152 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:02.956159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:02.956370 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:02.957207 (+ 837us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:02.957414 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.959227 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.962013 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.963865 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.964873 (+ 1008us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.965188 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:02.965283 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 12:51:02.965458 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":2177,"threads_started":1}
I20260924 12:51:02.966281 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:02.952405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.2.81.67:41931)
0924 12:51:02.954560 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:02.954570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:02.954605 (+ 35us) server_negotiation.cc:407] Connection header received
0924 12:51:02.954705 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:02.954712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:02.954995 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:02.955189 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:02.955987 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.958221 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.961686 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:02.963739 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:02.963891 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:02.965651 (+ 1760us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:02.965734 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 12:51:02.965921 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1999,"thread_start_us":1854,"threads_started":1}
I20260924 12:51:02.967690 20337 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6736c16e8a4c64a78fc80731a69534 (DEFAULT_TABLE table=test-table [id=cb3a1056d92246c09ccb89c4b0981d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:02.969231 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6736c16e8a4c64a78fc80731a69534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:02.969152 20467 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6736c16e8a4c64a78fc80731a69534 (DEFAULT_TABLE table=test-table [id=cb3a1056d92246c09ccb89c4b0981d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:02.969539 20206 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6736c16e8a4c64a78fc80731a69534 (DEFAULT_TABLE table=test-table [id=cb3a1056d92246c09ccb89c4b0981d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:02.970477 20467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6736c16e8a4c64a78fc80731a69534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:02.970813 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6736c16e8a4c64a78fc80731a69534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:02.986065 20550 tablet_bootstrap.cc:492] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Bootstrap starting.
I20260924 12:51:02.989336 20552 tablet_bootstrap.cc:492] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Bootstrap starting.
I20260924 12:51:02.990528 20551 tablet_bootstrap.cc:492] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Bootstrap starting.
I20260924 12:51:02.991257 20550 tablet_bootstrap.cc:654] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:02.992740 20550 log.cc:824] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:02.993206 20552 tablet_bootstrap.cc:654] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:02.993918 20551 tablet_bootstrap.cc:654] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:02.994774 20552 log.cc:824] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:02.995096 20551 log.cc:824] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:02.997697 20550 tablet_bootstrap.cc:492] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: No bootstrap required, opened a new log
I20260924 12:51:02.997673 20271 heartbeater.cc:503] Master 127.2.81.126:45723 was elected leader, sending a full tablet report...
I20260924 12:51:02.998037 20550 ts_tablet_manager.cc:1397] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260924 12:51:02.998101 20551 tablet_bootstrap.cc:492] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: No bootstrap required, opened a new log
I20260924 12:51:02.998366 20551 ts_tablet_manager.cc:1397] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260924 12:51:02.999519 20552 tablet_bootstrap.cc:492] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: No bootstrap required, opened a new log
I20260924 12:51:02.999825 20552 ts_tablet_manager.cc:1397] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260924 12:51:03.008132 20550 raft_consensus.cc:348] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.008905 20550 raft_consensus.cc:374] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:03.009049 20550 raft_consensus.cc:729] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e10d1be30949cca143d724c4e6901d, State: Initialized, Role: FOLLOWER
I20260924 12:51:03.009747 20550 consensus_queue.cc:260] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.010859 20551 raft_consensus.cc:348] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.011328 20551 raft_consensus.cc:374] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:03.011420 20551 raft_consensus.cc:729] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d54cdbc5a640a8b691a48e088fd754, State: Initialized, Role: FOLLOWER
I20260924 12:51:03.011581 20550 ts_tablet_manager.cc:1428] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 12:51:03.012022 20552 raft_consensus.cc:348] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.012188 20551 consensus_queue.cc:260] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.012480 20552 raft_consensus.cc:374] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:03.012588 20552 raft_consensus.cc:729] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74cb8af4f5c405eac0e0ca1f083730b, State: Initialized, Role: FOLLOWER
I20260924 12:51:03.013325 20552 consensus_queue.cc:260] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.014693 20532 heartbeater.cc:503] Master 127.2.81.126:45723 was elected leader, sending a full tablet report...
I20260924 12:51:03.015244 20551 ts_tablet_manager.cc:1428] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260924 12:51:03.016669 20402 heartbeater.cc:503] Master 127.2.81.126:45723 was elected leader, sending a full tablet report...
I20260924 12:51:03.018342 20552 ts_tablet_manager.cc:1428] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
I20260924 12:51:03.126613 20557 raft_consensus.cc:482] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:51:03.126904 20557 raft_consensus.cc:504] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.128584 20557 leader_election.cc:288] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d74cb8af4f5c405eac0e0ca1f083730b (127.2.81.66:40787), 96e10d1be30949cca143d724c4e6901d (127.2.81.65:40593)
I20260924 12:51:03.134268 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.129147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:50481 (local address 127.2.81.66:40787)
0924 12:51:03.129344 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:03.129350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:03.129372 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:51:03.129467 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:03.129473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:03.129558 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:03.129650 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:03.130153 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.130880 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.132622 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.133387 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.133546 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.133637 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:03.133890 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 12:51:03.134011 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 12:51:03.134286 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.129027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:40787 (local address 127.2.81.67:50481)
0924 12:51:03.129183 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.129202 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:03.129345 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:03.129711 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:03.129717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:03.129761 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:03.129984 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:03.129994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.131021 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.131027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:03.132472 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:03.132484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.133574 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.133588 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:03.133739 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.133767 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 12:51:03.133846 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 12:51:03.133991 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260924 12:51:03.134977 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.129179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:45363 (local address 127.2.81.65:40593)
0924 12:51:03.129362 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:03.129368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:03.130066 (+ 698us) server_negotiation.cc:407] Connection header received
0924 12:51:03.130251 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:03.130256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:03.130326 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:03.130400 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:03.130889 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.131634 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.133317 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.134077 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.134191 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.134287 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:03.134680 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 12:51:03.134785 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 12:51:03.135073 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.129027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.2.81.67:45363)
0924 12:51:03.129968 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.129981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:03.130126 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:03.130426 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:03.130430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:03.130453 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:03.130733 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:03.130742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.131771 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.131775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:03.133150 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:03.133162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.134238 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.134244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:03.134474 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.134508 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:51:03.134632 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 12:51:03.134761 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":230,"thread_start_us":381,"threads_started":1}
I20260924 12:51:03.135512 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6736c16e8a4c64a78fc80731a69534" candidate_uuid: "80d54cdbc5a640a8b691a48e088fd754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" is_pre_election: true
I20260924 12:51:03.136101 20357 raft_consensus.cc:2454] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d54cdbc5a640a8b691a48e088fd754 in term 0.
I20260924 12:51:03.136183 20226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6736c16e8a4c64a78fc80731a69534" candidate_uuid: "80d54cdbc5a640a8b691a48e088fd754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e10d1be30949cca143d724c4e6901d" is_pre_election: true
I20260924 12:51:03.136721 20226 raft_consensus.cc:2454] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d54cdbc5a640a8b691a48e088fd754 in term 0.
I20260924 12:51:03.136896 20423 leader_election.cc:302] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d54cdbc5a640a8b691a48e088fd754, d74cb8af4f5c405eac0e0ca1f083730b; no voters:
I20260924 12:51:03.137385 20557 raft_consensus.cc:2781] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:51:03.137504 20557 raft_consensus.cc:482] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:51:03.137609 20557 raft_consensus.cc:3037] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:03.139827 20557 raft_consensus.cc:504] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.140492 20557 leader_election.cc:288] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [CANDIDATE]: Term 1 election: Requested vote from peers d74cb8af4f5c405eac0e0ca1f083730b (127.2.81.66:40787), 96e10d1be30949cca143d724c4e6901d (127.2.81.65:40593)
I20260924 12:51:03.140960 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6736c16e8a4c64a78fc80731a69534" candidate_uuid: "80d54cdbc5a640a8b691a48e088fd754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74cb8af4f5c405eac0e0ca1f083730b"
I20260924 12:51:03.141115 20226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6736c16e8a4c64a78fc80731a69534" candidate_uuid: "80d54cdbc5a640a8b691a48e088fd754" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e10d1be30949cca143d724c4e6901d"
I20260924 12:51:03.141213 20357 raft_consensus.cc:3037] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:03.141323 20226 raft_consensus.cc:3037] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:03.143564 20226 raft_consensus.cc:2454] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d54cdbc5a640a8b691a48e088fd754 in term 1.
I20260924 12:51:03.143572 20357 raft_consensus.cc:2454] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d54cdbc5a640a8b691a48e088fd754 in term 1.
I20260924 12:51:03.143975 20422 leader_election.cc:302] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d54cdbc5a640a8b691a48e088fd754, 96e10d1be30949cca143d724c4e6901d; no voters:
I20260924 12:51:03.144290 20557 raft_consensus.cc:2781] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 12:51:03.144404 20533 tablet.cc:2369] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:03.144892 20557 raft_consensus.cc:686] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 1 LEADER]: Becoming Leader. State: Replica: 80d54cdbc5a640a8b691a48e088fd754, State: Running, Role: LEADER
I20260924 12:51:03.145399 20557 consensus_queue.cc:237] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } }
I20260924 12:51:03.149916 20085 catalog_manager.cc:5495] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80d54cdbc5a640a8b691a48e088fd754 (127.2.81.67). New cstate: current_term: 1 leader_uuid: "80d54cdbc5a640a8b691a48e088fd754" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d54cdbc5a640a8b691a48e088fd754" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 } health_report { overall_health: UNKNOWN } } }
W20260924 12:51:03.181775 20403 tablet.cc:2369] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:03.210348 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.203495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.2.81.67:41931)
0924 12:51:03.203735 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:03.203742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:03.203766 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:51:03.203879 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:03.203885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:03.203996 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:03.204178 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:03.204501 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.205718 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.207695 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.208363 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.208495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.209013 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:03.209714 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:03.209852 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:03.209974 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 12:51:03.210069 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 12:51:03.220276 20357 raft_consensus.cc:1261] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 1 FOLLOWER]: Refusing update from remote peer 80d54cdbc5a640a8b691a48e088fd754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:03.220687 20226 raft_consensus.cc:1261] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 1 FOLLOWER]: Refusing update from remote peer 80d54cdbc5a640a8b691a48e088fd754: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:03.221318 20557 consensus_queue.cc:1035] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74cb8af4f5c405eac0e0ca1f083730b" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:03.221709 20560 consensus_queue.cc:1035] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e10d1be30949cca143d724c4e6901d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 12:51:03.224196 20272 tablet.cc:2369] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:03.232265 20563 mvcc.cc:205] Tried to move back new op lower bound from 7332881462139449344 to 7332881461847220224. Current Snapshot: MvccSnapshot[applied={T|T < 7332881462139449344}]
I20260924 12:51:03.235221 20564 mvcc.cc:205] Tried to move back new op lower bound from 7332881462139449344 to 7332881461847220224. Current Snapshot: MvccSnapshot[applied={T|T < 7332881462139449344}]
I20260924 12:51:03.246372 20083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:03.246713 20083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:03.250658 20083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b11c81e35e8643d2ae81243409013480: Sending DeleteTablet for 3 replicas of tablet 2a6736c16e8a4c64a78fc80731a69534
I20260924 12:51:03.251682 20337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6736c16e8a4c64a78fc80731a69534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:03 UTC) from {username='slave'} at 127.0.0.1:45850
I20260924 12:51:03.251816 20467 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6736c16e8a4c64a78fc80731a69534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:03 UTC) from {username='slave'} at 127.0.0.1:43004
I20260924 12:51:03.252070 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6736c16e8a4c64a78fc80731a69534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:03 UTC) from {username='slave'} at 127.0.0.1:55940
I20260924 12:51:03.252628 20573 tablet_replica.cc:330] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: stopping tablet replica
I20260924 12:51:03.253146 20573 raft_consensus.cc:2229] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:03.253643 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20144
I20260924 12:51:03.253808 20573 raft_consensus.cc:2258] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:03.256126 20573 ts_tablet_manager.cc:1905] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:03.263859 20573 ts_tablet_manager.cc:1918] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:03.264281 20573 log.cc:1196] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/2a6736c16e8a4c64a78fc80731a69534
I20260924 12:51:03.264966 20573 ts_tablet_manager.cc:1939] T 2a6736c16e8a4c64a78fc80731a69534 P d74cb8af4f5c405eac0e0ca1f083730b: Deleting consensus metadata
I20260924 12:51:03.268042 20072 catalog_manager.cc:4841] TS d74cb8af4f5c405eac0e0ca1f083730b (127.2.81.66:40787): tablet 2a6736c16e8a4c64a78fc80731a69534 (table test-table [id=cb3a1056d92246c09ccb89c4b0981d26]) successfully deleted
I20260924 12:51:03.269485 20574 tablet_replica.cc:330] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: stopping tablet replica
I20260924 12:51:03.269938 20574 raft_consensus.cc:2229] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:51:03.270470 20575 tablet_replica.cc:330] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: stopping tablet replica
I20260924 12:51:03.270594 20574 raft_consensus.cc:2258] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:03.270859 20575 raft_consensus.cc:2229] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:03.271354 20575 raft_consensus.cc:2258] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:03.272513 20574 ts_tablet_manager.cc:1905] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:03.273939 20575 ts_tablet_manager.cc:1905] T 2a6736c16e8a4c64a78fc80731a69534 P 96e10d1be30949cca143d724c4e6901d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:03.274252 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.255640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.2.81.65:40593)
0924 12:51:03.255822 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:03.255828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:03.264851 (+ 9023us) server_negotiation.cc:407] Connection header received
0924 12:51:03.265379 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:03.265386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:03.265510 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:03.265587 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:03.266300 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.267606 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.269742 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.270421 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.270567 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.272415 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:03.272426 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:03.272443 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:03.273224 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:03.273282 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:03.273307 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:03.273418 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:03.273730 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 12:51:03.273852 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:51:03.276018 20266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:03.280560 20574 ts_tablet_manager.cc:1918] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:03.280781 20574 log.cc:1196] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/2a6736c16e8a4c64a78fc80731a69534
I20260924 12:51:03.281363 20574 ts_tablet_manager.cc:1939] T 2a6736c16e8a4c64a78fc80731a69534 P 80d54cdbc5a640a8b691a48e088fd754: Deleting consensus metadata
I20260924 12:51:03.283376 20071 catalog_manager.cc:4841] TS 80d54cdbc5a640a8b691a48e088fd754 (127.2.81.67:41931): tablet 2a6736c16e8a4c64a78fc80731a69534 (table test-table [id=cb3a1056d92246c09ccb89c4b0981d26]) successfully deleted
I20260924 12:51:03.370378 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20144
W20260924 12:51:03.388916 20071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:40593 (error 108)
I20260924 12:51:03.390251 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20275
I20260924 12:51:03.390784 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.390282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55962)
0924 12:51:03.390505 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.390611 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 12:51:03.391073 20071 catalog_manager.cc:4584] TS 96e10d1be30949cca143d724c4e6901d (127.2.81.65:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a6736c16e8a4c64a78fc80731a69534: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
I20260924 12:51:03.398206 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.391172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.2.81.66:40787)
0924 12:51:03.391367 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:03.391374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:03.391479 (+ 105us) server_negotiation.cc:407] Connection header received
0924 12:51:03.391647 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:03.391651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:03.391732 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:03.391814 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:03.392282 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.393120 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.395003 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.395468 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.395578 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.396640 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:03.396648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:03.396663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:03.397388 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:03.397437 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:03.397460 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:03.397560 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:03.397699 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 12:51:03.397818 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:51:03.399093 20397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:03.445102 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.444416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55968)
0924 12:51:03.444673 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.444789 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 12:51:03.446434 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.446005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55972)
0924 12:51:03.446170 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.446280 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 12:51:03.489538 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20275
I20260924 12:51:03.503310 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20405
I20260924 12:51:03.506892 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.506413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55980)
0924 12:51:03.506612 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.506711 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 12:51:03.508003 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.507617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55996)
0924 12:51:03.507762 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.507838 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 12:51:03.511399 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.504247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.2.81.67:41931)
0924 12:51:03.504479 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:03.504485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:03.504503 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:51:03.504566 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:03.504570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:03.504645 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:03.504739 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:03.505193 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.506013 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.507575 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:03.508292 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:03.508434 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:03.509713 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:03.509726 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:03.509749 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:03.510558 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:03.510645 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:03.510671 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:03.510783 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:03.510946 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 12:51:03.511055 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:51:03.512122 20527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:03.588729 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.588183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:55998)
0924 12:51:03.588427 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.588549 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 12:51:03.589793 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:03.589377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40593 (local address 127.0.0.1:56002)
0924 12:51:03.589602 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:03.589677 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 12:51:03.631763 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20405
I20260924 12:51:03.646564 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20054
I20260924 12:51:03.647715 20115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:03.759704 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20054
2026-09-24T12:51:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2912 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:03 dist-test-slave-9tm1 krb5kdc[20584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:03 dist-test-slave-9tm1 krb5kdc[20584](info): set up 2 sockets
Sep 24 12:51:03 dist-test-slave-9tm1 krb5kdc[20584](info): commencing operation
krb5kdc: starting...
W20260924 12:51:04.833169 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:04 dist-test-slave-9tm1 krb5kdc[20584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:05.085912 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39363
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:05.417187 20597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:05.417493 20597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:05.417610 20597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:05.425236 20597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:05.425359 20597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:05.425432 20597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:05.425506 20597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:05.425566 20597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:05.435693 20597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:05.437031 20597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:05.438330 20597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:05.450133 20602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:05.450170 20603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:05.450747 20605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:05.451195 20597 server_base.cc:1034] running on GCE node
I20260924 12:51:05.451953 20597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:05.453475 20597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:05.454747 20597 hybrid_clock.cc:648] HybridClock initialized: now 1790254265454691 us; error 40 us; skew 500 ppm
I20260924 12:51:05.455190 20597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:05 dist-test-slave-9tm1 krb5kdc[20584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254265, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:05.458716 20597 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:05.460287 20597 webserver.cc:460] Webserver started at http://127.2.81.126:36061/ using document root <none> and password file <none>
I20260924 12:51:05.460999 20597 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:05.461124 20597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:05.461423 20597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:05.464407 20597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "481fd7293ef74e80a33a16ce43596468"
format_stamp: "Formatted at 2026-09-24 12:51:05 on dist-test-slave-9tm1"
server_key: "f5cc15a1a119b282e63e9873f38ce9a8"
server_key_iv: "f6b0a458444e884875c5a4582bed252e"
server_key_version: "encryptionkey@0"
I20260924 12:51:05.465225 20597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "481fd7293ef74e80a33a16ce43596468"
format_stamp: "Formatted at 2026-09-24 12:51:05 on dist-test-slave-9tm1"
server_key: "f5cc15a1a119b282e63e9873f38ce9a8"
server_key_iv: "f6b0a458444e884875c5a4582bed252e"
server_key_version: "encryptionkey@0"
I20260924 12:51:05.470499 20597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:51:05.474347 20612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:05.475795 20597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 12:51:05.475975 20597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "481fd7293ef74e80a33a16ce43596468"
format_stamp: "Formatted at 2026-09-24 12:51:05 on dist-test-slave-9tm1"
server_key: "f5cc15a1a119b282e63e9873f38ce9a8"
server_key_iv: "f6b0a458444e884875c5a4582bed252e"
server_key_version: "encryptionkey@0"
I20260924 12:51:05.476178 20597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:05.496390 20597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:05.500716 20597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:05.501024 20597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:05.517946 20597 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:39363
I20260924 12:51:05.517972 20663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:39363 every 8 connection(s)
I20260924 12:51:05.519289 20597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:05.519734 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20597
I20260924 12:51:05.519896 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:05.520437 2373 external_mini_cluster.cc:1428] Setting key dfe63f8b8b3398a8cc14b259d9a6c382
I20260924 12:51:05.523483 20664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:51:05 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:05.534933 20664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468: Bootstrap starting.
I20260924 12:51:05.538607 20664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:05.539896 20664 log.cc:824] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:05.543279 20664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468: No bootstrap required, opened a new log
I20260924 12:51:05.543484 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:05.522178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.2.81.126:39363)
0924 12:51:05.522898 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:05.522910 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:05.522947 (+ 37us) server_negotiation.cc:407] Connection header received
0924 12:51:05.524649 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:05.524670 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:05.525106 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:05.525452 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:05.526957 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:05.528329 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:05.530132 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:05.530841 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:05.531017 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:05.535426 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:05.535436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:05.535460 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:05.535506 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:05.538951 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:05.539847 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:05.539854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:05.539860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:05.540102 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:05.540549 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:05.540554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:05.540557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:05.541383 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:05.541617 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:05.542075 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 12:51:05.542416 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":312,"threads_started":1}
I20260924 12:51:05.551406 20664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481fd7293ef74e80a33a16ce43596468" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39363 } }
I20260924 12:51:05.551807 20664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:05.551910 20664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481fd7293ef74e80a33a16ce43596468, State: Initialized, Role: FOLLOWER
I20260924 12:51:05.552443 20664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481fd7293ef74e80a33a16ce43596468" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39363 } }
I20260924 12:51:05.552659 20664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:05.552778 20664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:05.552947 20664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:05.555352 20664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481fd7293ef74e80a33a16ce43596468" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39363 } }
I20260924 12:51:05.555857 20664 leader_election.cc:302] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481fd7293ef74e80a33a16ce43596468; no voters:
I20260924 12:51:05.556465 20664 leader_election.cc:288] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:05.556589 20669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:05.557304 20669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [term 1 LEADER]: Becoming Leader. State: Replica: 481fd7293ef74e80a33a16ce43596468, State: Running, Role: LEADER
I20260924 12:51:05.557819 20669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481fd7293ef74e80a33a16ce43596468" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39363 } }
I20260924 12:51:05.558549 20664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:05.560323 20670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481fd7293ef74e80a33a16ce43596468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481fd7293ef74e80a33a16ce43596468" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39363 } } }
I20260924 12:51:05.560753 20670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:05.560437 20671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481fd7293ef74e80a33a16ce43596468. Latest consensus state: current_term: 1 leader_uuid: "481fd7293ef74e80a33a16ce43596468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481fd7293ef74e80a33a16ce43596468" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39363 } } }
I20260924 12:51:05.561371 20671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:05.562515 20677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:05.567303 20677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:05.576956 20677 catalog_manager.cc:1347] Generated new cluster ID: 4139de166cc846fd8e6e64ec603785ba
I20260924 12:51:05.577128 20677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:05.598167 20677 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:05.600154 20677 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:05.614662 20677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468: Generated new TSK 0
I20260924 12:51:05.615790 20677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:51:05.685369 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39363
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:06.016726 20690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:06.017069 20690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:06.017218 20690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:06.024763 20690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:06.024885 20690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:06.025074 20690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:51:06.035315 20690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:06.036628 20690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:06.038012 20690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:06.051674 20695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:06.051698 20696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:06.051718 20698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:06.052244 20690 server_base.cc:1034] running on GCE node
I20260924 12:51:06.053057 20690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:06.054055 20690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:06.055298 20690 hybrid_clock.cc:648] HybridClock initialized: now 1790254266055241 us; error 42 us; skew 500 ppm
I20260924 12:51:06.055639 20690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:06 dist-test-slave-9tm1 krb5kdc[20584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254266, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:06.059103 20690 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:51:06.060652 20690 webserver.cc:460] Webserver started at http://127.2.81.65:34799/ using document root <none> and password file <none>
I20260924 12:51:06.061394 20690 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:06.061513 20690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:06.061805 20690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:06.064828 20690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "8bd2ef96cbeb488b85ff09b23f83a404"
format_stamp: "Formatted at 2026-09-24 12:51:06 on dist-test-slave-9tm1"
server_key: "4f2bef25437204c487956d1a0c23a403"
server_key_iv: "05b4afbbcfb07393dba59e0878cd28fe"
server_key_version: "encryptionkey@0"
I20260924 12:51:06.065655 20690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "8bd2ef96cbeb488b85ff09b23f83a404"
format_stamp: "Formatted at 2026-09-24 12:51:06 on dist-test-slave-9tm1"
server_key: "4f2bef25437204c487956d1a0c23a403"
server_key_iv: "05b4afbbcfb07393dba59e0878cd28fe"
server_key_version: "encryptionkey@0"
I20260924 12:51:06.071139 20690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 12:51:06.074767 20705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:06.076237 20690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:51:06.076406 20690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "8bd2ef96cbeb488b85ff09b23f83a404"
format_stamp: "Formatted at 2026-09-24 12:51:06 on dist-test-slave-9tm1"
server_key: "4f2bef25437204c487956d1a0c23a403"
server_key_iv: "05b4afbbcfb07393dba59e0878cd28fe"
server_key_version: "encryptionkey@0"
I20260924 12:51:06.076618 20690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:06.109405 20690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:06.113914 20690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:06.114205 20690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:06.115141 20690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:06.116500 20690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:06.116582 20690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:06.116709 20690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:06.116768 20690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:06.147039 20690 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:32819
I20260924 12:51:06.147112 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:32819 every 8 connection(s)
I20260924 12:51:06.148294 20690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:51:06.149271 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20690
I20260924 12:51:06.149428 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:51:06.150012 2373 external_mini_cluster.cc:1428] Setting key 6501c50f69582eeeadbf473026098e29
Sep 24 12:51:06 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254266, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:06.171382 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:06.151676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:45065 (local address 127.2.81.126:39363)
0924 12:51:06.152165 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:06.152172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:06.153281 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 12:51:06.155196 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:06.155201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:06.155281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:06.155393 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:06.157049 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.158000 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.159832 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.160420 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.160534 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:06.165162 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:06.165168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:06.165172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:06.165210 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:06.167833 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:06.168845 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:06.168850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:06.168853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:06.169040 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:06.169556 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:06.169559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:06.169561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:06.169856 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:06.169993 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:06.170920 (+ 927us) server_negotiation.cc:299] Negotiation successful
0924 12:51:06.171123 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":300,"threads_started":1}
I20260924 12:51:06.172880 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:06.152293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39363 (local address 127.2.81.65:45065)
0924 12:51:06.153116 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:06.153162 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:06.154923 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:06.155663 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:06.155673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:06.156225 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:06.156845 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:06.156862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.158180 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.158184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:06.159661 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:06.159693 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.160700 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.160707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:06.161031 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:06.161805 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:06.161833 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:06.164853 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:06.168016 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:06.168027 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:06.168049 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:06.168680 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:06.169171 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:06.169176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:06.169179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:06.169439 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:06.169984 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:06.169992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:06.170479 (+ 487us) client_negotiation.cc:769] Sending connection context
0924 12:51:06.170815 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 12:51:06.171399 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":329,"threads_started":1}
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:51:06.174726 20818 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39363
I20260924 12:51:06.175033 20818 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:06.175840 20818 heartbeater.cc:511] Master 127.2.81.126:39363 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:51:06.178365 20629 ts_manager.cc:194] Registered new tserver with Master: 8bd2ef96cbeb488b85ff09b23f83a404 (127.2.81.65:32819)
I20260924 12:51:06.180342 20629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:45065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:51:06.209556 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39363
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:06.539685 20824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:06.540056 20824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:06.540207 20824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:06.547737 20824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:06.547866 20824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:06.547977 20824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:51:06.557889 20824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:06.559172 20824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:06.560477 20824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:06.574376 20829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:06.574296 20830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:06.574309 20832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:06.578802 20824 server_base.cc:1034] running on GCE node
I20260924 12:51:06.579439 20824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:06.580538 20824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:06.581794 20824 hybrid_clock.cc:648] HybridClock initialized: now 1790254266581736 us; error 52 us; skew 500 ppm
I20260924 12:51:06.582078 20824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:06 dist-test-slave-9tm1 krb5kdc[20584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254266, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:06.585309 20824 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:51:06.586766 20824 webserver.cc:460] Webserver started at http://127.2.81.66:37157/ using document root <none> and password file <none>
I20260924 12:51:06.587476 20824 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:06.587617 20824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:06.587906 20824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:06.590799 20824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "e7e9c20fbf2345bca38bbbcab8e6d490"
format_stamp: "Formatted at 2026-09-24 12:51:06 on dist-test-slave-9tm1"
server_key: "603e27aacdbaa80d7a7529a7c090f55f"
server_key_iv: "c3022c7c4146d10e504523ba96b4dec9"
server_key_version: "encryptionkey@0"
I20260924 12:51:06.591544 20824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "e7e9c20fbf2345bca38bbbcab8e6d490"
format_stamp: "Formatted at 2026-09-24 12:51:06 on dist-test-slave-9tm1"
server_key: "603e27aacdbaa80d7a7529a7c090f55f"
server_key_iv: "c3022c7c4146d10e504523ba96b4dec9"
server_key_version: "encryptionkey@0"
I20260924 12:51:06.597481 20824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:51:06.601004 20839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:06.602375 20824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:06.602528 20824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "e7e9c20fbf2345bca38bbbcab8e6d490"
format_stamp: "Formatted at 2026-09-24 12:51:06 on dist-test-slave-9tm1"
server_key: "603e27aacdbaa80d7a7529a7c090f55f"
server_key_iv: "c3022c7c4146d10e504523ba96b4dec9"
server_key_version: "encryptionkey@0"
I20260924 12:51:06.602744 20824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:06.615167 20824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:06.619593 20824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:06.619880 20824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:06.620949 20824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:06.622416 20824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:06.622498 20824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:06.622617 20824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:06.622668 20824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:06.653719 20824 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:41333
I20260924 12:51:06.653882 20951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:41333 every 8 connection(s)
I20260924 12:51:06.654951 20824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:51:06.662907 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20824
I20260924 12:51:06.663074 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:51:06.663546 2373 external_mini_cluster.cc:1428] Setting key 4a140d80e7908227505f038deabadf75
Sep 24 12:51:06 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254266, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:06.676280 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:06.657452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44339 (local address 127.2.81.126:39363)
0924 12:51:06.657628 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:06.657632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:06.658689 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 12:51:06.660566 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:06.660572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:06.660651 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:06.660737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:06.662257 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.663259 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.665029 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.665608 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.665706 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:06.669944 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:06.669950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:06.669953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:06.669989 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:06.672520 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:06.673570 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:06.673574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:06.673577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:06.673755 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:06.674305 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:06.674310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:06.674313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:06.674704 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:06.674851 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:06.675818 (+ 967us) server_negotiation.cc:299] Negotiation successful
0924 12:51:06.676034 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:51:06.677649 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:06.657857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39363 (local address 127.2.81.66:44339)
0924 12:51:06.658541 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:06.658576 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:06.660299 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:06.660962 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:06.660992 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:06.661528 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:06.662089 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:06.662107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.663419 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.663424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:06.664856 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:06.664871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:06.665859 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:06.665867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:06.666184 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:06.666898 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:06.666931 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:06.669680 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:06.672691 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:06.672701 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:06.672719 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:06.673372 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:06.673910 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:06.673915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:06.673918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:06.674183 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:06.674864 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:06.674873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:06.675312 (+ 439us) client_negotiation.cc:769] Sending connection context
0924 12:51:06.675663 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 12:51:06.676216 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":271,"threads_started":1}
I20260924 12:51:06.679116 20952 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39363
I20260924 12:51:06.679419 20952 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:06.680204 20952 heartbeater.cc:511] Master 127.2.81.126:39363 requested a full tablet report, sending...
I20260924 12:51:06.681753 20629 ts_manager.cc:194] Registered new tserver with Master: e7e9c20fbf2345bca38bbbcab8e6d490 (127.2.81.66:41333)
I20260924 12:51:06.682610 20629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:44339
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:51:06.719830 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39363
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:07.048887 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:07.049242 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:07.049396 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:07.057538 20957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:07.057806 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:07.057997 20957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:51:07.069190 20957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:39363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:07.070533 20957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:07.071836 20957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:07.085225 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:07.085270 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:07.085177 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:07.085769 20957 server_base.cc:1034] running on GCE node
I20260924 12:51:07.086496 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:07.087497 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:07.088706 20957 hybrid_clock.cc:648] HybridClock initialized: now 1790254267088650 us; error 57 us; skew 500 ppm
I20260924 12:51:07.089059 20957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:07 dist-test-slave-9tm1 krb5kdc[20584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254267, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:07.092139 20957 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:51:07.093786 20957 webserver.cc:460] Webserver started at http://127.2.81.67:46409/ using document root <none> and password file <none>
I20260924 12:51:07.094494 20957 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:07.094606 20957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:07.094882 20957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:07.097831 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "a92eaab20a3f46c8931c1cc9f728f707"
format_stamp: "Formatted at 2026-09-24 12:51:07 on dist-test-slave-9tm1"
server_key: "3aeb771439dba44b5a21a9d7ead6fef3"
server_key_iv: "c318ebec142386ac143a35003b531835"
server_key_version: "encryptionkey@0"
I20260924 12:51:07.098639 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "a92eaab20a3f46c8931c1cc9f728f707"
format_stamp: "Formatted at 2026-09-24 12:51:07 on dist-test-slave-9tm1"
server_key: "3aeb771439dba44b5a21a9d7ead6fef3"
server_key_iv: "c318ebec142386ac143a35003b531835"
server_key_version: "encryptionkey@0"
I20260924 12:51:07.103772 20957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 12:51:07.107244 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:07.108654 20957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:07.108806 20957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "a92eaab20a3f46c8931c1cc9f728f707"
format_stamp: "Formatted at 2026-09-24 12:51:07 on dist-test-slave-9tm1"
server_key: "3aeb771439dba44b5a21a9d7ead6fef3"
server_key_iv: "c318ebec142386ac143a35003b531835"
server_key_version: "encryptionkey@0"
I20260924 12:51:07.109047 20957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:07.125463 20957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:07.130024 20957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:07.130306 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:07.131287 20957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:07.132644 20957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:07.132722 20957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:07.132846 20957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:07.132899 20957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:07.167582 20957 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:40253
I20260924 12:51:07.167663 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:40253 every 8 connection(s)
I20260924 12:51:07.168910 20957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:51:07.174422 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 20957
I20260924 12:51:07.174625 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:51:07.175148 2373 external_mini_cluster.cc:1428] Setting key 10c15d3e13f18e61700b83fdc0fcd4d9
I20260924 12:51:07.183678 20818 heartbeater.cc:503] Master 127.2.81.126:39363 was elected leader, sending a full tablet report...
Sep 24 12:51:07 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254267, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:07.191917 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.171617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:53731 (local address 127.2.81.126:39363)
0924 12:51:07.171818 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.171824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.173134 (+ 1310us) server_negotiation.cc:407] Connection header received
0924 12:51:07.175670 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.175676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.175778 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.175902 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:07.177929 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.178998 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.180990 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.181508 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.181618 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.185539 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:07.185545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:07.185549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:07.185584 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:07.188315 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.189245 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.189250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.189252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.189439 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.189931 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.189937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.189941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.190358 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:07.190515 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.191288 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.191563 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:51:07.192904 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.172082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39363 (local address 127.2.81.67:53731)
0924 12:51:07.172909 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:07.172959 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:07.175213 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:07.176292 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:07.176302 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:07.177023 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:07.177745 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.177764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.179202 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.179207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.180797 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.180812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.181716 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.181722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.181938 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.182668 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:07.182696 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:07.185298 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:07.188520 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:07.188528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:07.188544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:07.189099 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:07.189578 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:07.189583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:07.189586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:07.189806 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:07.190501 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:07.190510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:07.190897 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 12:51:07.191160 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 12:51:07.191741 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":356,"threads_started":1}
I20260924 12:51:07.194881 21085 heartbeater.cc:347] Connected to a master server at 127.2.81.126:39363
I20260924 12:51:07.195226 21085 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:07.196020 21085 heartbeater.cc:511] Master 127.2.81.126:39363 requested a full tablet report, sending...
I20260924 12:51:07.197718 20629 ts_manager.cc:194] Registered new tserver with Master: a92eaab20a3f46c8931c1cc9f728f707 (127.2.81.67:40253)
I20260924 12:51:07.198843 20629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:53731
I20260924 12:51:07.203506 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:51:07.221890 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.207907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.2.81.126:39363)
0924 12:51:07.208116 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.208121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.208399 (+ 278us) server_negotiation.cc:407] Connection header received
0924 12:51:07.208540 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.208544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.208616 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.208705 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:07.210157 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.210957 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.212429 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.212966 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.213113 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.215976 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:07.215982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:07.215986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:07.216063 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:07.218776 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.219780 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.219786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.219789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.219966 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.220456 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.220460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.220462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.220808 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:07.221138 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.221479 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.221651 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 12:51:07.226395 20629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:51:07.233479 20629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:51:07.259265 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.247077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.2.81.65:32819)
0924 12:51:07.248170 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.248179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.248889 (+ 710us) server_negotiation.cc:407] Connection header received
0924 12:51:07.249036 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.249044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.249327 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.249518 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:07.250516 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.252508 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.256420 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.257963 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.258105 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.258348 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.258726 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.258922 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":1059,"threads_started":1}
I20260924 12:51:07.259752 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.246476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:32819 (local address 127.0.0.1:36310)
0924 12:51:07.248263 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:07.248283 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:07.248426 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:07.249807 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:07.249815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:07.249849 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:07.250304 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.250317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.252740 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.252747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.256188 (+ 3441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.256207 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.258224 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.258231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.258448 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.258506 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 12:51:07.258668 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 12:51:07.258993 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"mutex_wait_us":70,"thread_start_us":1814,"threads_started":1}
I20260924 12:51:07.261919 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.246324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:41333 (local address 127.0.0.1:56996)
0924 12:51:07.247366 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:07.247406 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:07.247585 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:07.254186 (+ 6601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:07.254191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:07.254223 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:07.254851 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.254864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.257432 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.257491 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.259316 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.259333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.261249 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.261255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.261420 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.261466 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 12:51:07.261535 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 12:51:07.261646 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":71,"thread_start_us":665,"threads_started":1}
I20260924 12:51:07.262974 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.246514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.2.81.66:41333)
0924 12:51:07.253438 (+ 6924us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.253449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.253481 (+ 32us) server_negotiation.cc:407] Connection header received
0924 12:51:07.253591 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.253599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.253916 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.254390 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:07.255335 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.257291 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.259536 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.261031 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.262274 (+ 1243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.262423 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.262480 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.262640 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6735,"thread_start_us":6613,"threads_started":1}
I20260924 12:51:07.263465 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.248653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:40253 (local address 127.0.0.1:47914)
0924 12:51:07.254296 (+ 5643us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:07.254313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:07.254452 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:07.256020 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:07.256025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:07.256056 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:07.256421 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.256433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.258383 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.258388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.260795 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.260809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.262667 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.262673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.262841 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.262872 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 12:51:07.262971 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 12:51:07.263113 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5492,"mutex_wait_us":35,"thread_start_us":5334,"threads_started":1}
I20260924 12:51:07.263545 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.248926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.2.81.67:40253)
0924 12:51:07.253831 (+ 4905us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.253838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.254602 (+ 764us) server_negotiation.cc:407] Connection header received
0924 12:51:07.254715 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.254723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.255217 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.255630 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:07.256665 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.258184 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.261022 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.262474 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.262603 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.262800 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.263022 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.263187 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4738,"thread_start_us":4562,"threads_started":1}
I20260924 12:51:07.264349 20753 tablet_service.cc:1467] Processing CreateTablet for tablet e321955df6e0435a96d257ac6da032bb (DEFAULT_TABLE table=test-table [id=af1b96f9e40b4268bf4cc63b640778ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:07.266034 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e321955df6e0435a96d257ac6da032bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:07.266755 20887 tablet_service.cc:1467] Processing CreateTablet for tablet e321955df6e0435a96d257ac6da032bb (DEFAULT_TABLE table=test-table [id=af1b96f9e40b4268bf4cc63b640778ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:07.267833 20887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e321955df6e0435a96d257ac6da032bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:07.267663 21020 tablet_service.cc:1467] Processing CreateTablet for tablet e321955df6e0435a96d257ac6da032bb (DEFAULT_TABLE table=test-table [id=af1b96f9e40b4268bf4cc63b640778ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:07.268955 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e321955df6e0435a96d257ac6da032bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:07.278788 21103 tablet_bootstrap.cc:492] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Bootstrap starting.
I20260924 12:51:07.284382 21103 tablet_bootstrap.cc:654] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:07.284708 21105 tablet_bootstrap.cc:492] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Bootstrap starting.
I20260924 12:51:07.286147 21103 log.cc:824] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:07.286533 21104 tablet_bootstrap.cc:492] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Bootstrap starting.
I20260924 12:51:07.288481 21105 tablet_bootstrap.cc:654] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:07.289413 21104 tablet_bootstrap.cc:654] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:07.289744 21105 log.cc:824] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:07.290580 21104 log.cc:824] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:07.291720 21103 tablet_bootstrap.cc:492] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: No bootstrap required, opened a new log
I20260924 12:51:07.292034 21103 ts_tablet_manager.cc:1397] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260924 12:51:07.293167 21105 tablet_bootstrap.cc:492] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: No bootstrap required, opened a new log
I20260924 12:51:07.293423 21105 ts_tablet_manager.cc:1397] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:51:07.294021 21104 tablet_bootstrap.cc:492] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: No bootstrap required, opened a new log
I20260924 12:51:07.294261 21104 ts_tablet_manager.cc:1397] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 12:51:07.300911 21105 raft_consensus.cc:348] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.301463 21105 raft_consensus.cc:374] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:07.301584 21105 raft_consensus.cc:729] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92eaab20a3f46c8931c1cc9f728f707, State: Initialized, Role: FOLLOWER
I20260924 12:51:07.302366 21105 consensus_queue.cc:260] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.302736 21103 raft_consensus.cc:348] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.303213 21103 raft_consensus.cc:374] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:07.303309 21103 raft_consensus.cc:729] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9c20fbf2345bca38bbbcab8e6d490, State: Initialized, Role: FOLLOWER
I20260924 12:51:07.303748 21085 heartbeater.cc:503] Master 127.2.81.126:39363 was elected leader, sending a full tablet report...
I20260924 12:51:07.303897 21103 consensus_queue.cc:260] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.304486 21105 ts_tablet_manager.cc:1428] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 12:51:07.304898 21104 raft_consensus.cc:348] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.305383 21104 raft_consensus.cc:374] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:07.305488 21104 raft_consensus.cc:729] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd2ef96cbeb488b85ff09b23f83a404, State: Initialized, Role: FOLLOWER
I20260924 12:51:07.306157 21104 consensus_queue.cc:260] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.306625 21103 ts_tablet_manager.cc:1428] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 12:51:07.307057 20952 heartbeater.cc:503] Master 127.2.81.126:39363 was elected leader, sending a full tablet report...
I20260924 12:51:07.309099 21104 ts_tablet_manager.cc:1428] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260924 12:51:07.401388 20819 tablet.cc:2369] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:51:07.406888 20953 tablet.cc:2369] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:51:07.420590 21086 tablet.cc:2369] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:07.509974 21110 raft_consensus.cc:482] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:51:07.510268 21110 raft_consensus.cc:504] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.511878 21110 leader_election.cc:288] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bd2ef96cbeb488b85ff09b23f83a404 (127.2.81.65:32819), a92eaab20a3f46c8931c1cc9f728f707 (127.2.81.67:40253)
I20260924 12:51:07.518404 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.512332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:40779 (local address 127.2.81.65:32819)
0924 12:51:07.512538 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.512545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.513005 (+ 460us) server_negotiation.cc:407] Connection header received
0924 12:51:07.513230 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.513236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.513316 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.513419 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:07.514164 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.514880 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.516655 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.517475 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.517633 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.517739 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.518033 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.518163 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 12:51:07.518769 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.512232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:32819 (local address 127.2.81.66:40779)
0924 12:51:07.512858 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:07.512873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:07.513065 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:07.513605 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:07.513610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:07.513636 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:07.514034 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.514045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.515005 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.515010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.516492 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.516506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.517697 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.517703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.517869 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.517894 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:51:07.517975 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 12:51:07.518119 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":3,"thread_start_us":360,"threads_started":1}
I20260924 12:51:07.519572 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.513277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:36617 (local address 127.2.81.67:40253)
0924 12:51:07.513497 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.513502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.513862 (+ 360us) server_negotiation.cc:407] Connection header received
0924 12:51:07.514036 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.514040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.514101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.514187 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:07.514764 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.515478 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.517152 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.518193 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.518319 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.518417 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.519202 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.519342 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 12:51:07.519785 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.513034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:40253 (local address 127.2.81.66:36617)
0924 12:51:07.513735 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:07.513752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:07.513873 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:07.514227 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:07.514231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:07.514255 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:07.514591 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.514600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.515619 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.515624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.517005 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:07.517022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.518781 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.518789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:07.518991 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.519026 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:51:07.519149 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 12:51:07.519281 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":361,"threads_started":1}
I20260924 12:51:07.519960 20773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e321955df6e0435a96d257ac6da032bb" candidate_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" is_pre_election: true
I20260924 12:51:07.520469 20773 raft_consensus.cc:2454] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e9c20fbf2345bca38bbbcab8e6d490 in term 0.
I20260924 12:51:07.520645 21040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e321955df6e0435a96d257ac6da032bb" candidate_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a92eaab20a3f46c8931c1cc9f728f707" is_pre_election: true
I20260924 12:51:07.521191 20842 leader_election.cc:302] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bd2ef96cbeb488b85ff09b23f83a404, e7e9c20fbf2345bca38bbbcab8e6d490; no voters:
I20260924 12:51:07.521186 21040 raft_consensus.cc:2454] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7e9c20fbf2345bca38bbbcab8e6d490 in term 0.
I20260924 12:51:07.521610 21110 raft_consensus.cc:2781] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:51:07.521752 21110 raft_consensus.cc:482] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:51:07.521862 21110 raft_consensus.cc:3037] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:07.524008 21110 raft_consensus.cc:504] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.524590 21110 leader_election.cc:288] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [CANDIDATE]: Term 1 election: Requested vote from peers 8bd2ef96cbeb488b85ff09b23f83a404 (127.2.81.65:32819), a92eaab20a3f46c8931c1cc9f728f707 (127.2.81.67:40253)
I20260924 12:51:07.525197 21040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e321955df6e0435a96d257ac6da032bb" candidate_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a92eaab20a3f46c8931c1cc9f728f707"
I20260924 12:51:07.525211 20773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e321955df6e0435a96d257ac6da032bb" candidate_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd2ef96cbeb488b85ff09b23f83a404"
I20260924 12:51:07.525393 21040 raft_consensus.cc:3037] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:07.525426 20773 raft_consensus.cc:3037] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:07.527629 21040 raft_consensus.cc:2454] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e9c20fbf2345bca38bbbcab8e6d490 in term 1.
I20260924 12:51:07.528052 20842 leader_election.cc:302] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a92eaab20a3f46c8931c1cc9f728f707, e7e9c20fbf2345bca38bbbcab8e6d490; no voters:
I20260924 12:51:07.528323 21110 raft_consensus.cc:2781] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:07.528626 20773 raft_consensus.cc:2454] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7e9c20fbf2345bca38bbbcab8e6d490 in term 1.
I20260924 12:51:07.529103 21110 raft_consensus.cc:686] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 1 LEADER]: Becoming Leader. State: Replica: e7e9c20fbf2345bca38bbbcab8e6d490, State: Running, Role: LEADER
I20260924 12:51:07.529621 21110 consensus_queue.cc:237] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } }
I20260924 12:51:07.534047 20627 catalog_manager.cc:5495] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7e9c20fbf2345bca38bbbcab8e6d490 (127.2.81.66). New cstate: current_term: 1 leader_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9c20fbf2345bca38bbbcab8e6d490" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 41333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20260924 12:51:07.559512 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.553977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.2.81.66:41333)
0924 12:51:07.554186 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.554191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.554209 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:51:07.554338 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.554342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.554419 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.554541 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:07.554861 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.555550 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.557056 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.557449 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.557546 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.558174 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:07.558763 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:07.558901 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.559162 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.559218 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 12:51:07.567726 20773 raft_consensus.cc:1261] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 1 FOLLOWER]: Refusing update from remote peer e7e9c20fbf2345bca38bbbcab8e6d490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:07.567727 21040 raft_consensus.cc:1261] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 1 FOLLOWER]: Refusing update from remote peer e7e9c20fbf2345bca38bbbcab8e6d490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:07.568759 21114 consensus_queue.cc:1035] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a92eaab20a3f46c8931c1cc9f728f707" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:07.569121 21110 consensus_queue.cc:1035] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bd2ef96cbeb488b85ff09b23f83a404" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:07.584431 21119 mvcc.cc:205] Tried to move back new op lower bound from 7332881479947227136 to 7332881479805181952. Current Snapshot: MvccSnapshot[applied={T|T < 7332881479947227136}]
I20260924 12:51:07.598934 20628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:07.599321 20628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:07.603495 20628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 481fd7293ef74e80a33a16ce43596468: Sending DeleteTablet for 3 replicas of tablet e321955df6e0435a96d257ac6da032bb
I20260924 12:51:07.604743 20887 tablet_service.cc:1514] Processing DeleteTablet for tablet e321955df6e0435a96d257ac6da032bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:07 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:56996
I20260924 12:51:07.604892 20753 tablet_service.cc:1514] Processing DeleteTablet for tablet e321955df6e0435a96d257ac6da032bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:07 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:36310
I20260924 12:51:07.604892 21020 tablet_service.cc:1514] Processing DeleteTablet for tablet e321955df6e0435a96d257ac6da032bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:07 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:47914
I20260924 12:51:07.609163 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20690
I20260924 12:51:07.623088 21129 tablet_replica.cc:330] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: stopping tablet replica
I20260924 12:51:07.623591 21129 raft_consensus.cc:2229] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:07.623637 21130 tablet_replica.cc:330] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: stopping tablet replica
I20260924 12:51:07.624042 21130 raft_consensus.cc:2229] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:07.624146 21129 raft_consensus.cc:2258] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:07.624538 21130 raft_consensus.cc:2258] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:07.625178 21128 tablet_replica.cc:330] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: stopping tablet replica
I20260924 12:51:07.626595 21129 ts_tablet_manager.cc:1905] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:07.626619 21130 ts_tablet_manager.cc:1905] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:07.628268 21128 raft_consensus.cc:2229] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:51:07.631316 21128 raft_consensus.cc:2258] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:07.633957 21129 ts_tablet_manager.cc:1918] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:07.634021 21130 ts_tablet_manager.cc:1918] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:07.634150 21129 log.cc:1196] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-0/wal/wals/e321955df6e0435a96d257ac6da032bb
I20260924 12:51:07.634209 21130 log.cc:1196] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/e321955df6e0435a96d257ac6da032bb
I20260924 12:51:07.634722 21130 ts_tablet_manager.cc:1939] T e321955df6e0435a96d257ac6da032bb P a92eaab20a3f46c8931c1cc9f728f707: Deleting consensus metadata
Sep 24 12:51:07 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:51:07.635361 21129 ts_tablet_manager.cc:1939] T e321955df6e0435a96d257ac6da032bb P 8bd2ef96cbeb488b85ff09b23f83a404: Deleting consensus metadata
I20260924 12:51:07.636554 21128 ts_tablet_manager.cc:1905] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:07.636713 20615 catalog_manager.cc:4841] TS a92eaab20a3f46c8931c1cc9f728f707 (127.2.81.67:40253): tablet e321955df6e0435a96d257ac6da032bb (table test-table [id=af1b96f9e40b4268bf4cc63b640778ed]) successfully deleted
I20260924 12:51:07.637123 20615 catalog_manager.cc:4841] TS 8bd2ef96cbeb488b85ff09b23f83a404 (127.2.81.65:32819): tablet e321955df6e0435a96d257ac6da032bb (table test-table [id=af1b96f9e40b4268bf4cc63b640778ed]) successfully deleted
I20260924 12:51:07.642567 21128 ts_tablet_manager.cc:1918] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:07.642773 21128 log.cc:1196] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/e321955df6e0435a96d257ac6da032bb
I20260924 12:51:07.643362 21128 ts_tablet_manager.cc:1939] T e321955df6e0435a96d257ac6da032bb P e7e9c20fbf2345bca38bbbcab8e6d490: Deleting consensus metadata
I20260924 12:51:07.645093 20614 catalog_manager.cc:4841] TS e7e9c20fbf2345bca38bbbcab8e6d490 (127.2.81.66:41333): tablet e321955df6e0435a96d257ac6da032bb (table test-table [id=af1b96f9e40b4268bf4cc63b640778ed]) successfully deleted
I20260924 12:51:07.645852 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.610806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.2.81.65:32819)
0924 12:51:07.611026 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.611034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.623420 (+ 12386us) server_negotiation.cc:407] Connection header received
0924 12:51:07.623747 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.623752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.623858 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.624012 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:07.625880 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.626968 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.629001 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.629645 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.629752 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.638167 (+ 8415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:07.638184 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:07.638202 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:07.638250 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:07.642332 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.643371 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.643382 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.643385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.643569 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.644020 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.644026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.644029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.644523 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:07.644844 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.645219 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.645468 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 12:51:07.646652 20813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:07.752557 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20690
I20260924 12:51:07.774139 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20824
Sep 24 12:51:07 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:51:07.791683 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.774881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.2.81.66:41333)
0924 12:51:07.775085 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.775092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.775114 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:51:07.775193 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.775199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.775285 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.775417 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:07.776781 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.778116 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.779563 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.780277 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.780438 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.785111 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:07.785126 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:07.785145 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:07.785204 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:07.788185 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.789384 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.789392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.789394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.789561 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.790004 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.790008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.790010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.790390 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:07.790744 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.791193 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.791381 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:51:07.792884 20947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:07.908689 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20824
I20260924 12:51:07.924547 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20957
Sep 24 12:51:07 dist-test-slave-9tm1 krb5kdc[20584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:51:07.939941 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:07.925347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.2.81.67:40253)
0924 12:51:07.925544 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:07.925550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:07.925571 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:51:07.925738 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:07.925742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:07.925805 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:07.925888 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:07.927210 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.927923 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.929271 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:07.929710 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:07.929819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:07.934441 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:07.934457 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:07.934474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:07.934519 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:07.937218 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.938014 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.938022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.938024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.938194 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:07.938570 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:07.938573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:07.938575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:07.938908 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:07.939145 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:07.939480 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 12:51:07.939670 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 12:51:07.940704 21080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:08.030289 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20957
I20260924 12:51:08.043293 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 20597
I20260924 12:51:08.044364 20659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:08.157094 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 20597
2026-09-24T12:51:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4398 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T12:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:08Z Disabled control of system clock
I20260924 12:51:08.230034 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:42983
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 12:51:08.567097 21139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:08.567416 21139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:08.567502 21139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:08.575057 21139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:08.575181 21139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:08.575218 21139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:08.575277 21139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:08.575321 21139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:08.584913 21139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:42983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:42983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:08.586261 21139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:08.588102 21139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:08.601994 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:08.607928 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:08.607928 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:08.608027 21139 server_base.cc:1034] running on GCE node
I20260924 12:51:08.609107 21139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:08.610734 21139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:08.611987 21139 hybrid_clock.cc:648] HybridClock initialized: now 1790254268611937 us; error 38 us; skew 500 ppm
I20260924 12:51:08.612404 21139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:08.615031 21139 webserver.cc:460] Webserver started at http://127.2.81.126:34837/ using document root <none> and password file <none>
I20260924 12:51:08.615880 21139 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:08.616029 21139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:08.616434 21139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:08.619637 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "cf778c877a6d4cb3aef871bd9b387180"
format_stamp: "Formatted at 2026-09-24 12:51:08 on dist-test-slave-9tm1"
server_key: "58cbed01ae952c1e4deeff7bec0261fc"
server_key_iv: "069e8a8490f8c40c8c15cffbdb13abbd"
server_key_version: "encryptionkey@0"
I20260924 12:51:08.620469 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "cf778c877a6d4cb3aef871bd9b387180"
format_stamp: "Formatted at 2026-09-24 12:51:08 on dist-test-slave-9tm1"
server_key: "58cbed01ae952c1e4deeff7bec0261fc"
server_key_iv: "069e8a8490f8c40c8c15cffbdb13abbd"
server_key_version: "encryptionkey@0"
I20260924 12:51:08.626266 21139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:51:08.629990 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:08.631436 21139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:51:08.631655 21139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "cf778c877a6d4cb3aef871bd9b387180"
format_stamp: "Formatted at 2026-09-24 12:51:08 on dist-test-slave-9tm1"
server_key: "58cbed01ae952c1e4deeff7bec0261fc"
server_key_iv: "069e8a8490f8c40c8c15cffbdb13abbd"
server_key_version: "encryptionkey@0"
I20260924 12:51:08.631875 21139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:08.660305 21139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:08.661208 21139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:08.661477 21139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:08.678833 21139 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:42983
I20260924 12:51:08.678851 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:42983 every 8 connection(s)
I20260924 12:51:08.680222 21139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:08.684082 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21139
I20260924 12:51:08.684192 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:08.684257 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:08.684687 2373 external_mini_cluster.cc:1428] Setting key 72e1c72b84bf063467c4d551c6284bd6
I20260924 12:51:08.697817 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180: Bootstrap starting.
I20260924 12:51:08.698642 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:08.686674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.2.81.126:42983)
0924 12:51:08.687788 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:08.687809 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:08.687863 (+ 54us) server_negotiation.cc:407] Connection header received
0924 12:51:08.689529 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:08.689557 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:08.690079 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:08.690486 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:08.691104 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:08.692439 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:08.693957 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:08.694694 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:08.694896 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:08.696089 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:08.696098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:08.696114 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:08.696677 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:08.696768 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:08.696791 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:08.696897 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:08.697175 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 12:51:08.697535 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":386,"threads_started":1}
I20260924 12:51:08.701419 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:08.702458 21205 log.cc:824] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:08.705250 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180: No bootstrap required, opened a new log
I20260924 12:51:08.712692 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf778c877a6d4cb3aef871bd9b387180" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42983 } }
I20260924 12:51:08.713162 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:08.713263 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf778c877a6d4cb3aef871bd9b387180, State: Initialized, Role: FOLLOWER
I20260924 12:51:08.713863 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf778c877a6d4cb3aef871bd9b387180" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42983 } }
I20260924 12:51:08.714110 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:08.714238 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:08.714388 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:08.716753 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf778c877a6d4cb3aef871bd9b387180" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42983 } }
I20260924 12:51:08.717305 21205 leader_election.cc:302] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf778c877a6d4cb3aef871bd9b387180; no voters:
I20260924 12:51:08.717851 21205 leader_election.cc:288] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:08.717988 21210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:08.718712 21210 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [term 1 LEADER]: Becoming Leader. State: Replica: cf778c877a6d4cb3aef871bd9b387180, State: Running, Role: LEADER
I20260924 12:51:08.719269 21210 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf778c877a6d4cb3aef871bd9b387180" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42983 } }
I20260924 12:51:08.719913 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:08.722090 21211 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf778c877a6d4cb3aef871bd9b387180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf778c877a6d4cb3aef871bd9b387180" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42983 } } }
I20260924 12:51:08.722508 21211 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:08.723004 21212 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf778c877a6d4cb3aef871bd9b387180. Latest consensus state: current_term: 1 leader_uuid: "cf778c877a6d4cb3aef871bd9b387180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf778c877a6d4cb3aef871bd9b387180" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 42983 } } }
I20260924 12:51:08.723394 21212 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:08.724762 21219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:08.729986 21219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:08.739543 21219 catalog_manager.cc:1347] Generated new cluster ID: ab2a6848e05b47648930984a32566547
I20260924 12:51:08.739691 21219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:08.758041 21219 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:08.759838 21219 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:08.778040 21219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180: Generated new TSK 0
I20260924 12:51:08.779299 21219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:08.788192 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42983
--builtin_ntp_servers=127.2.81.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 12:51:09.250424 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:09.250728 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:09.250995 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:09.259589 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:09.259740 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:09.259887 21229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:51:09.269627 21229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:09.270999 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:09.272796 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:09.289351 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:09.289278 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:09.289312 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:09.290200 21229 server_base.cc:1034] running on GCE node
I20260924 12:51:09.291091 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:09.292173 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:09.293473 21229 hybrid_clock.cc:648] HybridClock initialized: now 1790254269293419 us; error 47 us; skew 500 ppm
I20260924 12:51:09.293902 21229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:09.300650 21229 webserver.cc:460] Webserver started at http://127.2.81.65:43661/ using document root <none> and password file <none>
I20260924 12:51:09.301549 21229 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:09.301668 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:09.301997 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:09.305063 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "661ed50937704cac83d795d761f3996d"
format_stamp: "Formatted at 2026-09-24 12:51:09 on dist-test-slave-9tm1"
server_key: "3ae04bceb65d63171c658e2ef1e2e356"
server_key_iv: "b71551406a4c70fee4a15f573b92c4a6"
server_key_version: "encryptionkey@0"
I20260924 12:51:09.305856 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "661ed50937704cac83d795d761f3996d"
format_stamp: "Formatted at 2026-09-24 12:51:09 on dist-test-slave-9tm1"
server_key: "3ae04bceb65d63171c658e2ef1e2e356"
server_key_iv: "b71551406a4c70fee4a15f573b92c4a6"
server_key_version: "encryptionkey@0"
I20260924 12:51:09.311935 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 12:51:09.315934 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:09.317648 21229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:51:09.317830 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "661ed50937704cac83d795d761f3996d"
format_stamp: "Formatted at 2026-09-24 12:51:09 on dist-test-slave-9tm1"
server_key: "3ae04bceb65d63171c658e2ef1e2e356"
server_key_iv: "b71551406a4c70fee4a15f573b92c4a6"
server_key_version: "encryptionkey@0"
I20260924 12:51:09.318077 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:09.336035 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:09.336946 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:09.337262 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:09.338248 21229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:09.339836 21229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:09.339939 21229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:09.340052 21229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:09.340130 21229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:09.373533 21229 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:40215
I20260924 12:51:09.373605 21355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:40215 every 8 connection(s)
I20260924 12:51:09.374866 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:51:09.383057 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21229
I20260924 12:51:09.383273 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:51:09.383809 2373 external_mini_cluster.cc:1428] Setting key 10ca61e49c77493d364fa404dbc8c97c
I20260924 12:51:09.386873 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42983
--builtin_ntp_servers=127.2.81.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 12:51:09.389297 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:09.377640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:34217 (local address 127.2.81.126:42983)
0924 12:51:09.378151 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:09.378155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:09.379381 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 12:51:09.381309 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:09.381316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:09.381594 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:09.381729 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:09.382998 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.383811 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.385975 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.386407 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.386511 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:09.388295 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:09.388299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:09.388302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:09.388322 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:09.388368 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:09.388382 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:09.388460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:09.388741 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 12:51:09.388922 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260924 12:51:09.390636 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:09.378205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:42983 (local address 127.2.81.65:34217)
0924 12:51:09.379193 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:09.379242 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:09.380990 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:09.382033 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:09.382043 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:09.382149 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:09.382780 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:09.382801 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.384014 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.384020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:09.385805 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:09.385822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.386664 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.386673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:09.387003 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:09.387830 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:09.387834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:09.387894 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:09.388015 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:09.388480 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:09.388491 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:09.388498 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:51:09.388688 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 12:51:09.389285 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":341,"threads_started":1}
I20260924 12:51:09.392442 21356 heartbeater.cc:347] Connected to a master server at 127.2.81.126:42983
I20260924 12:51:09.392766 21356 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:09.393699 21356 heartbeater.cc:511] Master 127.2.81.126:42983 requested a full tablet report, sending...
I20260924 12:51:09.396441 21170 ts_manager.cc:194] Registered new tserver with Master: 661ed50937704cac83d795d761f3996d (127.2.81.65:40215)
I20260924 12:51:09.398581 21170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.65:34217
W20260924 12:51:09.723840 21360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:09.724135 21360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:09.724272 21360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:09.731820 21360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:09.731941 21360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:09.732080 21360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:51:09.743077 21360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:09.744414 21360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:09.746233 21360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:09.759573 21365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:09.759629 21366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:09.759675 21368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:09.760427 21360 server_base.cc:1034] running on GCE node
I20260924 12:51:09.761252 21360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:09.762324 21360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:09.763545 21360 hybrid_clock.cc:648] HybridClock initialized: now 1790254269763507 us; error 42 us; skew 500 ppm
I20260924 12:51:09.763898 21360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:09.766666 21360 webserver.cc:460] Webserver started at http://127.2.81.66:33651/ using document root <none> and password file <none>
I20260924 12:51:09.767452 21360 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:09.767560 21360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:09.767885 21360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:09.771085 21360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "bfa7a8d75fe24dee9855155ef6645cb6"
format_stamp: "Formatted at 2026-09-24 12:51:09 on dist-test-slave-9tm1"
server_key: "18400550471e535c3f411dc9786a363e"
server_key_iv: "8eba672dfe79322ed24a67034d793a31"
server_key_version: "encryptionkey@0"
I20260924 12:51:09.771979 21360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "bfa7a8d75fe24dee9855155ef6645cb6"
format_stamp: "Formatted at 2026-09-24 12:51:09 on dist-test-slave-9tm1"
server_key: "18400550471e535c3f411dc9786a363e"
server_key_iv: "8eba672dfe79322ed24a67034d793a31"
server_key_version: "encryptionkey@0"
I20260924 12:51:09.777748 21360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 12:51:09.782127 21374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:09.783759 21360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:09.783951 21360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "bfa7a8d75fe24dee9855155ef6645cb6"
format_stamp: "Formatted at 2026-09-24 12:51:09 on dist-test-slave-9tm1"
server_key: "18400550471e535c3f411dc9786a363e"
server_key_iv: "8eba672dfe79322ed24a67034d793a31"
server_key_version: "encryptionkey@0"
I20260924 12:51:09.784190 21360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:09.807847 21360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:09.808773 21360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:09.809115 21360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:09.810161 21360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:09.811962 21360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:09.812067 21360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:09.812183 21360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:09.812253 21360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:09.844216 21360 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:45205
I20260924 12:51:09.844307 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:45205 every 8 connection(s)
I20260924 12:51:09.845886 21360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:51:09.855892 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21360
I20260924 12:51:09.856053 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:51:09.856629 2373 external_mini_cluster.cc:1428] Setting key 326a2f7a6d347976156b37e352401c14
I20260924 12:51:09.860127 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42983
--builtin_ntp_servers=127.2.81.84:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 12:51:09.860316 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:09.848668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:35181 (local address 127.2.81.126:42983)
0924 12:51:09.848890 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:09.848896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:09.850403 (+ 1507us) server_negotiation.cc:407] Connection header received
0924 12:51:09.852381 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:09.852387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:09.852576 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:09.852709 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:09.854118 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.854886 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.856700 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.857218 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.857342 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:09.859386 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:09.859393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:09.859398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:09.859421 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:09.859469 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:09.859486 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:09.859590 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:09.859859 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 12:51:09.859990 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 12:51:09.861554 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:09.849213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:42983 (local address 127.2.81.66:35181)
0924 12:51:09.850211 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:09.850257 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:09.852038 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:09.853055 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:09.853066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:09.853186 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:09.853890 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:09.853913 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.855075 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.855079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:09.856515 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:09.856533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:09.857470 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:09.857478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:09.857921 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:09.858928 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:09.858932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:09.858998 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:09.859151 (+ 153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:09.859617 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:09.859627 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:09.859634 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:51:09.859821 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 12:51:09.860224 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":320,"threads_started":1}
I20260924 12:51:09.863061 21487 heartbeater.cc:347] Connected to a master server at 127.2.81.126:42983
I20260924 12:51:09.863360 21487 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:09.864225 21487 heartbeater.cc:511] Master 127.2.81.126:42983 requested a full tablet report, sending...
I20260924 12:51:09.865932 21170 ts_manager.cc:194] Registered new tserver with Master: bfa7a8d75fe24dee9855155ef6645cb6 (127.2.81.66:45205)
I20260924 12:51:09.866822 21170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.66:35181
W20260924 12:51:10.197981 21490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:10.198283 21490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:10.198400 21490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:10.206908 21490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:10.207046 21490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:10.207211 21490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:51:10.216758 21490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:42983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:10.218082 21490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:10.219853 21490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:10.233361 21495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:10.233407 21498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:10.233448 21496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:10.234277 21490 server_base.cc:1034] running on GCE node
I20260924 12:51:10.234915 21490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:10.235912 21490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:10.237093 21490 hybrid_clock.cc:648] HybridClock initialized: now 1790254270237021 us; error 59 us; skew 500 ppm
I20260924 12:51:10.237437 21490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:10.239895 21490 webserver.cc:460] Webserver started at http://127.2.81.67:38577/ using document root <none> and password file <none>
I20260924 12:51:10.240622 21490 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:10.240731 21490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:10.241082 21490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:10.243983 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "dafd32d529584cf2a399b27d6e86c225"
format_stamp: "Formatted at 2026-09-24 12:51:10 on dist-test-slave-9tm1"
server_key: "9e10ca9c2d449bea4eadcc0441341bda"
server_key_iv: "d3ae4c3b7bfe2b5e33c6b15ebf15bdb3"
server_key_version: "encryptionkey@0"
I20260924 12:51:10.244820 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "dafd32d529584cf2a399b27d6e86c225"
format_stamp: "Formatted at 2026-09-24 12:51:10 on dist-test-slave-9tm1"
server_key: "9e10ca9c2d449bea4eadcc0441341bda"
server_key_iv: "d3ae4c3b7bfe2b5e33c6b15ebf15bdb3"
server_key_version: "encryptionkey@0"
I20260924 12:51:10.250874 21490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 12:51:10.255185 21504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:10.257109 21490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:51:10.257310 21490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "dafd32d529584cf2a399b27d6e86c225"
format_stamp: "Formatted at 2026-09-24 12:51:10 on dist-test-slave-9tm1"
server_key: "9e10ca9c2d449bea4eadcc0441341bda"
server_key_iv: "d3ae4c3b7bfe2b5e33c6b15ebf15bdb3"
server_key_version: "encryptionkey@0"
I20260924 12:51:10.257584 21490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:10.276929 21490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:10.277935 21490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:10.278189 21490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:10.279165 21490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:10.280789 21490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:10.280879 21490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:10.280959 21490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:10.281055 21490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 12:51:10.312824 21490 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:43593
I20260924 12:51:10.312873 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:43593 every 8 connection(s)
I20260924 12:51:10.314141 21490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:51:10.315452 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21490
I20260924 12:51:10.315637 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:51:10.316095 2373 external_mini_cluster.cc:1428] Setting key b43ae0b6076eb1c06487e62e6b1e31f0
I20260924 12:51:10.326902 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.316897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:53547 (local address 127.2.81.126:42983)
0924 12:51:10.317138 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.317144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.318603 (+ 1459us) server_negotiation.cc:407] Connection header received
0924 12:51:10.320487 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.320493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.320583 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.320671 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:10.321763 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.322504 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.324066 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.324488 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.324622 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.326074 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:10.326079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:10.326082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:10.326101 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:10.326132 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:10.326143 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:10.326216 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.326433 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.326535 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 12:51:10.327950 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.317454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:42983 (local address 127.2.81.67:53547)
0924 12:51:10.318464 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.318502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:10.320198 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:10.320899 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:10.320908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:10.321032 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:10.321590 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.321607 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.322651 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.322654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.323929 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.323943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.324724 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.324731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.324955 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.325754 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:10.325757 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:10.325799 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:10.325888 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:10.326227 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:10.326238 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:10.326246 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:51:10.326393 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 12:51:10.326858 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":316,"threads_started":1}
I20260924 12:51:10.329447 21617 heartbeater.cc:347] Connected to a master server at 127.2.81.126:42983
I20260924 12:51:10.329768 21617 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:10.330585 21617 heartbeater.cc:511] Master 127.2.81.126:42983 requested a full tablet report, sending...
I20260924 12:51:10.332015 21170 ts_manager.cc:194] Registered new tserver with Master: dafd32d529584cf2a399b27d6e86c225 (127.2.81.67:43593)
I20260924 12:51:10.332774 21170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.67:53547
I20260924 12:51:10.344112 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:51:10.354451 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.348346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.2.81.126:42983)
0924 12:51:10.348583 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.348590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.348857 (+ 267us) server_negotiation.cc:407] Connection header received
0924 12:51:10.349065 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.349068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.349133 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.349215 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:10.349697 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.350422 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.351831 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.352250 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.352361 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.353592 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:10.353597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:10.353600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:10.353621 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:10.353657 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:10.353670 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:10.353838 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.354127 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.354184 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:51:10.359423 21170 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:51:10.366519 21170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:51:10.402426 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.379481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:45205 (local address 127.0.0.1:35550)
0924 12:51:10.392303 (+ 12822us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.392327 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:10.392494 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:10.393350 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:10.393358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:10.393393 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:10.393852 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.393866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.396886 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.396893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.399468 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.399483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.401398 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.401405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.401645 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.401706 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 12:51:10.401851 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 12:51:10.402041 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12652,"mutex_wait_us":2,"thread_start_us":19675,"threads_started":1}
I20260924 12:51:10.402587 21356 heartbeater.cc:503] Master 127.2.81.126:42983 was elected leader, sending a full tablet report...
I20260924 12:51:10.403250 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.379865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.2.81.66:45205)
0924 12:51:10.389357 (+ 9492us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.389369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.392640 (+ 3271us) server_negotiation.cc:407] Connection header received
0924 12:51:10.392766 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.392775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.393100 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.393317 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:10.394097 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.396710 (+ 2613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.399689 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.401202 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.401348 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.401604 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.401898 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.402104 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9308,"thread_start_us":9184,"threads_started":1}
I20260924 12:51:10.405328 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.379350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43593 (local address 127.0.0.1:54212)
0924 12:51:10.391346 (+ 11996us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.391410 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:10.391639 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:10.394968 (+ 3329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:10.394974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:10.395003 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:10.395473 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.395486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.400005 (+ 4519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.400011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.402200 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.402215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.404464 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.404475 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.404667 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.404702 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:51:10.404789 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 12:51:10.404922 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11688,"mutex_wait_us":31,"thread_start_us":14187,"threads_started":1}
I20260924 12:51:10.406260 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.379881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.2.81.67:43593)
0924 12:51:10.394304 (+ 14423us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.394314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.394353 (+ 39us) server_negotiation.cc:407] Connection header received
0924 12:51:10.394453 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.394462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.394721 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.395700 (+ 979us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:10.396935 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.398664 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.402518 (+ 3854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.403919 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.404051 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.404313 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.405727 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.405911 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14226,"thread_start_us":14056,"threads_started":1}
I20260924 12:51:10.407545 21422 tablet_service.cc:1467] Processing CreateTablet for tablet 4058f829d180410fabcf334a9c5584a3 (DEFAULT_TABLE table=test-table [id=0a9b05dc31e445a698ec72e09e8940df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:10.408835 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.379821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56844)
0924 12:51:10.399902 (+ 20081us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.399920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:10.400036 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:10.400762 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:10.400766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:10.400789 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:10.401276 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.401290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.403158 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.403162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.405622 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.405637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.407084 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.407091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.408232 (+ 1141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.408265 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 12:51:10.408353 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 12:51:10.408497 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19323,"mutex_wait_us":51,"thread_start_us":12044,"threads_started":1}
I20260924 12:51:10.408910 21422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4058f829d180410fabcf334a9c5584a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:10.409801 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.380381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.2.81.65:40215)
0924 12:51:10.399889 (+ 19508us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.399898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.400163 (+ 265us) server_negotiation.cc:407] Connection header received
0924 12:51:10.400274 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.400282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.400565 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.400758 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:10.401491 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.403007 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.405803 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.406887 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.407015 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.407273 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.408460 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.408633 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19322,"thread_start_us":19203,"threads_started":1}
I20260924 12:51:10.409866 21552 tablet_service.cc:1467] Processing CreateTablet for tablet 4058f829d180410fabcf334a9c5584a3 (DEFAULT_TABLE table=test-table [id=0a9b05dc31e445a698ec72e09e8940df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:10.411229 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4058f829d180410fabcf334a9c5584a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:10.413389 21291 tablet_service.cc:1467] Processing CreateTablet for tablet 4058f829d180410fabcf334a9c5584a3 (DEFAULT_TABLE table=test-table [id=0a9b05dc31e445a698ec72e09e8940df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:10.414852 21291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4058f829d180410fabcf334a9c5584a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:10.422593 21635 tablet_bootstrap.cc:492] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Bootstrap starting.
I20260924 12:51:10.424787 21636 tablet_bootstrap.cc:492] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Bootstrap starting.
I20260924 12:51:10.428414 21636 tablet_bootstrap.cc:654] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:10.428414 21635 tablet_bootstrap.cc:654] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:10.429728 21636 log.cc:824] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:10.429728 21635 log.cc:824] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:10.435312 21637 tablet_bootstrap.cc:492] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: Bootstrap starting.
I20260924 12:51:10.436599 21636 tablet_bootstrap.cc:492] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: No bootstrap required, opened a new log
I20260924 12:51:10.436928 21636 ts_tablet_manager.cc:1397] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260924 12:51:10.438524 21635 tablet_bootstrap.cc:492] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: No bootstrap required, opened a new log
I20260924 12:51:10.438827 21635 ts_tablet_manager.cc:1397] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260924 12:51:10.438952 21637 tablet_bootstrap.cc:654] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:10.440006 21637 log.cc:824] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:10.443030 21637 tablet_bootstrap.cc:492] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: No bootstrap required, opened a new log
I20260924 12:51:10.443493 21637 ts_tablet_manager.cc:1397] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 12:51:10.448310 21636 raft_consensus.cc:348] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.448841 21636 raft_consensus.cc:374] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:10.448940 21636 raft_consensus.cc:729] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa7a8d75fe24dee9855155ef6645cb6, State: Initialized, Role: FOLLOWER
I20260924 12:51:10.449676 21636 consensus_queue.cc:260] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.449641 21635 raft_consensus.cc:348] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.450083 21635 raft_consensus.cc:374] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:10.450188 21635 raft_consensus.cc:729] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafd32d529584cf2a399b27d6e86c225, State: Initialized, Role: FOLLOWER
I20260924 12:51:10.450872 21635 consensus_queue.cc:260] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.452097 21617 heartbeater.cc:503] Master 127.2.81.126:42983 was elected leader, sending a full tablet report...
I20260924 12:51:10.452867 21635 ts_tablet_manager.cc:1428] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 12:51:10.453840 21487 heartbeater.cc:503] Master 127.2.81.126:42983 was elected leader, sending a full tablet report...
I20260924 12:51:10.454438 21636 ts_tablet_manager.cc:1428] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260924 12:51:10.455811 21637 raft_consensus.cc:348] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.456298 21637 raft_consensus.cc:374] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:10.456401 21637 raft_consensus.cc:729] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ed50937704cac83d795d761f3996d, State: Initialized, Role: FOLLOWER
I20260924 12:51:10.457129 21637 consensus_queue.cc:260] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.459548 21637 ts_tablet_manager.cc:1428] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260924 12:51:10.565856 21618 tablet.cc:2369] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:10.566197 21641 raft_consensus.cc:482] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:51:10.566509 21641 raft_consensus.cc:504] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.568229 21641 leader_election.cc:288] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dafd32d529584cf2a399b27d6e86c225 (127.2.81.67:43593), 661ed50937704cac83d795d761f3996d (127.2.81.65:40215)
I20260924 12:51:10.574951 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.568669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:44803 (local address 127.2.81.65:40215)
0924 12:51:10.568873 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.568880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.569634 (+ 754us) server_negotiation.cc:407] Connection header received
0924 12:51:10.569795 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.569799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.569871 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.569945 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:10.570590 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.571270 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.573272 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.574058 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.574173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.574273 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.574579 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.574702 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:51:10.574988 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.568680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:38089 (local address 127.2.81.67:43593)
0924 12:51:10.568878 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.568884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.569526 (+ 642us) server_negotiation.cc:407] Connection header received
0924 12:51:10.569736 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.569741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.569801 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.569875 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:10.570524 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.571232 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.573287 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.574073 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.574186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.574300 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.574636 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.574760 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 12:51:10.575330 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.568694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.2.81.66:44803)
0924 12:51:10.569525 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.569539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:10.569670 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:10.569977 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:10.569981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:10.570007 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:10.570442 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.570451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.571405 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.571409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.573077 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.573090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.574267 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.574273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.574420 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.574460 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 12:51:10.574537 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 12:51:10.574676 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":72,"thread_start_us":439,"threads_started":1}
I20260924 12:51:10.575372 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.568613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43593 (local address 127.2.81.66:38089)
0924 12:51:10.569412 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.569428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:10.569586 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:10.569905 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:10.569910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:10.569932 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:10.570381 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.570390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.571355 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.571360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.573080 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:10.573093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.574309 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.574317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:10.574474 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.574504 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 12:51:10.574578 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 12:51:10.574700 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":105,"spinlock_wait_cycles":23424,"thread_start_us":474,"threads_started":1}
I20260924 12:51:10.576449 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4058f829d180410fabcf334a9c5584a3" candidate_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafd32d529584cf2a399b27d6e86c225" is_pre_election: true
I20260924 12:51:10.576448 21311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4058f829d180410fabcf334a9c5584a3" candidate_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ed50937704cac83d795d761f3996d" is_pre_election: true
I20260924 12:51:10.577054 21311 raft_consensus.cc:2454] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa7a8d75fe24dee9855155ef6645cb6 in term 0.
I20260924 12:51:10.577066 21572 raft_consensus.cc:2454] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa7a8d75fe24dee9855155ef6645cb6 in term 0.
I20260924 12:51:10.577826 21378 leader_election.cc:302] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 661ed50937704cac83d795d761f3996d, bfa7a8d75fe24dee9855155ef6645cb6, dafd32d529584cf2a399b27d6e86c225; no voters:
I20260924 12:51:10.578199 21641 raft_consensus.cc:2781] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:51:10.578344 21641 raft_consensus.cc:482] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:51:10.578442 21641 raft_consensus.cc:3037] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:10.580528 21641 raft_consensus.cc:504] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.581173 21641 leader_election.cc:288] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [CANDIDATE]: Term 1 election: Requested vote from peers dafd32d529584cf2a399b27d6e86c225 (127.2.81.67:43593), 661ed50937704cac83d795d761f3996d (127.2.81.65:40215)
I20260924 12:51:10.581632 21572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4058f829d180410fabcf334a9c5584a3" candidate_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafd32d529584cf2a399b27d6e86c225"
I20260924 12:51:10.581725 21311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4058f829d180410fabcf334a9c5584a3" candidate_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ed50937704cac83d795d761f3996d"
I20260924 12:51:10.581810 21572 raft_consensus.cc:3037] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:10.581951 21311 raft_consensus.cc:3037] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:10.583987 21572 raft_consensus.cc:2454] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa7a8d75fe24dee9855155ef6645cb6 in term 1.
I20260924 12:51:10.584134 21311 raft_consensus.cc:2454] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa7a8d75fe24dee9855155ef6645cb6 in term 1.
I20260924 12:51:10.584554 21378 leader_election.cc:302] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfa7a8d75fe24dee9855155ef6645cb6, dafd32d529584cf2a399b27d6e86c225; no voters:
I20260924 12:51:10.584897 21641 raft_consensus.cc:2781] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:10.585405 21641 raft_consensus.cc:686] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 1 LEADER]: Becoming Leader. State: Replica: bfa7a8d75fe24dee9855155ef6645cb6, State: Running, Role: LEADER
I20260924 12:51:10.586035 21641 consensus_queue.cc:237] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } }
I20260924 12:51:10.591033 21170 catalog_manager.cc:5495] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfa7a8d75fe24dee9855155ef6645cb6 (127.2.81.66). New cstate: current_term: 1 leader_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa7a8d75fe24dee9855155ef6645cb6" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 } health_report { overall_health: UNKNOWN } } }
W20260924 12:51:10.598033 21488 tablet.cc:2369] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:51:10.627354 21357 tablet.cc:2369] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:10.644431 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.639277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.2.81.66:45205)
0924 12:51:10.639443 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.639447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.639465 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:51:10.639549 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.639552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.639676 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.639803 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:10.640184 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.640948 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.642326 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.642780 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.642876 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.643359 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:10.643892 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:10.644026 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.644175 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.644215 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 12:51:10.652803 21311 raft_consensus.cc:1261] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 1 FOLLOWER]: Refusing update from remote peer bfa7a8d75fe24dee9855155ef6645cb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:10.652803 21572 raft_consensus.cc:1261] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 1 FOLLOWER]: Refusing update from remote peer bfa7a8d75fe24dee9855155ef6645cb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:10.654060 21646 consensus_queue.cc:1035] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dafd32d529584cf2a399b27d6e86c225" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:10.654450 21641 consensus_queue.cc:1035] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ed50937704cac83d795d761f3996d" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:10.686414 21169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:10.686712 21169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:10.690678 21169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf778c877a6d4cb3aef871bd9b387180: Sending DeleteTablet for 3 replicas of tablet 4058f829d180410fabcf334a9c5584a3
I20260924 12:51:10.691998 21552 tablet_service.cc:1514] Processing DeleteTablet for tablet 4058f829d180410fabcf334a9c5584a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:10 UTC) from {username='slave'} at 127.0.0.1:54212
I20260924 12:51:10.692838 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21229
I20260924 12:51:10.692776 21291 tablet_service.cc:1514] Processing DeleteTablet for tablet 4058f829d180410fabcf334a9c5584a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:10 UTC) from {username='slave'} at 127.0.0.1:56844
I20260924 12:51:10.696626 21422 tablet_service.cc:1514] Processing DeleteTablet for tablet 4058f829d180410fabcf334a9c5584a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:10 UTC) from {username='slave'} at 127.0.0.1:35550
I20260924 12:51:10.701738 21659 tablet_replica.cc:330] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: stopping tablet replica
I20260924 12:51:10.702283 21659 raft_consensus.cc:2229] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:10.702894 21659 raft_consensus.cc:2258] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:10.706257 21659 ts_tablet_manager.cc:1905] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:10.713060 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.693900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.2.81.65:40215)
0924 12:51:10.694086 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.694092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.703774 (+ 9682us) server_negotiation.cc:407] Connection header received
0924 12:51:10.703964 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.703971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.704089 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.704204 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:10.704940 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.706212 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.708553 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.709304 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.709411 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.711099 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:10.711113 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:10.711135 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:10.712000 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:10.712073 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:10.712103 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:10.712197 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.712470 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.712624 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 12:51:10.714030 21351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:10.714049 21659 ts_tablet_manager.cc:1918] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:10.714145 21660 tablet_replica.cc:330] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d: stopping tablet replica
I20260924 12:51:10.714226 21659 log.cc:1196] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/4058f829d180410fabcf334a9c5584a3
I20260924 12:51:10.714769 21659 ts_tablet_manager.cc:1939] T 4058f829d180410fabcf334a9c5584a3 P dafd32d529584cf2a399b27d6e86c225: Deleting consensus metadata
I20260924 12:51:10.716735 21157 catalog_manager.cc:4841] TS dafd32d529584cf2a399b27d6e86c225 (127.2.81.67:43593): tablet 4058f829d180410fabcf334a9c5584a3 (table test-table [id=0a9b05dc31e445a698ec72e09e8940df]) successfully deleted
I20260924 12:51:10.721698 21662 tablet_replica.cc:330] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: stopping tablet replica
I20260924 12:51:10.722283 21662 raft_consensus.cc:2229] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:51:10.723114 21662 raft_consensus.cc:2258] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:10.726024 21662 ts_tablet_manager.cc:1905] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:10.734045 21662 ts_tablet_manager.cc:1918] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:10.734238 21662 log.cc:1196] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/4058f829d180410fabcf334a9c5584a3
I20260924 12:51:10.734822 21662 ts_tablet_manager.cc:1939] T 4058f829d180410fabcf334a9c5584a3 P bfa7a8d75fe24dee9855155ef6645cb6: Deleting consensus metadata
I20260924 12:51:10.736768 21156 catalog_manager.cc:4841] TS bfa7a8d75fe24dee9855155ef6645cb6 (127.2.81.66:45205): tablet 4058f829d180410fabcf334a9c5584a3 (table test-table [id=0a9b05dc31e445a698ec72e09e8940df]) successfully deleted
I20260924 12:51:10.714620 21660 raft_consensus.cc:2229] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:10.813370 21660 raft_consensus.cc:2258] T 4058f829d180410fabcf334a9c5584a3 P 661ed50937704cac83d795d761f3996d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:10.813485 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21229
I20260924 12:51:10.827857 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21360
W20260924 12:51:10.827925 21155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:40215 (error 108)
I20260924 12:51:10.835119 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.834584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56862)
0924 12:51:10.834766 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.834909 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 12:51:10.835388 21155 catalog_manager.cc:4584] TS 661ed50937704cac83d795d761f3996d (127.2.81.65:40215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4058f829d180410fabcf334a9c5584a3: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
I20260924 12:51:10.837438 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.829003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.2.81.66:45205)
0924 12:51:10.829209 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.829216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.829239 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:51:10.829313 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.829319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.829428 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.829557 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:10.830016 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.831295 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.833171 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.833772 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.833890 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.835711 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:10.835722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:10.835741 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:10.836542 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:10.836603 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:10.836629 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:10.836750 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.836914 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.837044 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:51:10.838217 21482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:10.889240 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.888641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56872)
0924 12:51:10.888890 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.889044 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 12:51:10.890450 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.890039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56876)
0924 12:51:10.890205 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.890291 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 12:51:10.950738 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21360
I20260924 12:51:10.951071 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.950545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56882)
0924 12:51:10.950808 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.950910 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260924 12:51:10.952610 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.952166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56886)
0924 12:51:10.952355 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:10.952449 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 12:51:10.966010 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21490
I20260924 12:51:10.972584 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:10.966670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.2.81.67:43593)
0924 12:51:10.966826 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:10.966831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:10.966903 (+ 72us) server_negotiation.cc:407] Connection header received
0924 12:51:10.967052 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:10.967056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:10.967134 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:10.967220 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:10.967578 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.968306 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.969654 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:10.970049 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:10.970147 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:10.971240 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:10.971251 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:10.971266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:10.971862 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:10.971925 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:10.971948 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:10.972049 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:10.972206 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 12:51:10.972308 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 12:51:10.973421 21612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:11.033151 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:11.032610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56894)
0924 12:51:11.032823 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:11.032917 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 12:51:11.034200 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:11.033819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:40215 (local address 127.0.0.1:56904)
0924 12:51:11.034004 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:11.034093 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 12:51:11.060926 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21490
I20260924 12:51:11.073974 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21139
I20260924 12:51:11.075156 21200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:11.184693 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21139
2026-09-24T12:51:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3025 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:11 dist-test-slave-9tm1 krb5kdc[21670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:11 dist-test-slave-9tm1 krb5kdc[21670](info): set up 2 sockets
Sep 24 12:51:11 dist-test-slave-9tm1 krb5kdc[21670](info): commencing operation
krb5kdc: starting...
W20260924 12:51:12.221221 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:12 dist-test-slave-9tm1 krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:12.474189 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:38449
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:38449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:12.823802 21683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:12.824133 21683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:12.824271 21683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:12.831965 21683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:12.832110 21683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:12.832191 21683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:12.832267 21683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:12.832326 21683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:12.842657 21683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:38449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:38449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:12.844172 21683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:12.845547 21683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:12.858021 21688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:12.857951 21689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:12.857976 21691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:12.858623 21683 server_base.cc:1034] running on GCE node
I20260924 12:51:12.859560 21683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:12.861222 21683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:12.862521 21683 hybrid_clock.cc:648] HybridClock initialized: now 1790254272862385 us; error 81 us; skew 500 ppm
I20260924 12:51:12.862924 21683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:12 dist-test-slave-9tm1 krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254272, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:12.866295 21683 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:12.868192 21683 webserver.cc:460] Webserver started at http://127.2.81.126:39877/ using document root <none> and password file <none>
I20260924 12:51:12.868932 21683 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:12.869102 21683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:12.869413 21683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:12.872592 21683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "cc44d7ea157b4d849e7e4c6b31b56013"
format_stamp: "Formatted at 2026-09-24 12:51:12 on dist-test-slave-9tm1"
server_key: "20af1b0565d1fceb5e11ac4d689a2482"
server_key_iv: "943dc8b2a9947c90c9039a722fdf1bf7"
server_key_version: "encryptionkey@0"
I20260924 12:51:12.873441 21683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "cc44d7ea157b4d849e7e4c6b31b56013"
format_stamp: "Formatted at 2026-09-24 12:51:12 on dist-test-slave-9tm1"
server_key: "20af1b0565d1fceb5e11ac4d689a2482"
server_key_iv: "943dc8b2a9947c90c9039a722fdf1bf7"
server_key_version: "encryptionkey@0"
I20260924 12:51:12.879290 21683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:12.883193 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:12.884701 21683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:12.884932 21683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "cc44d7ea157b4d849e7e4c6b31b56013"
format_stamp: "Formatted at 2026-09-24 12:51:12 on dist-test-slave-9tm1"
server_key: "20af1b0565d1fceb5e11ac4d689a2482"
server_key_iv: "943dc8b2a9947c90c9039a722fdf1bf7"
server_key_version: "encryptionkey@0"
I20260924 12:51:12.885216 21683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:12.902020 21683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:12.906476 21683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:12.906809 21683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:12.926168 21683 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:38449
I20260924 12:51:12.926204 21749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:38449 every 8 connection(s)
I20260924 12:51:12.927551 21683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:12.929795 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21683
I20260924 12:51:12.929971 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:12.930473 2373 external_mini_cluster.cc:1428] Setting key 0a85312f4ffbd6c1743b866742b00ea8
I20260924 12:51:12.931728 21750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:51:12 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:12.944649 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013: Bootstrap starting.
I20260924 12:51:12.947913 21750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:12.949011 21750 log.cc:824] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:12.951884 21750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013: No bootstrap required, opened a new log
I20260924 12:51:12.952924 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:12.932258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.2.81.126:38449)
0924 12:51:12.933013 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:12.933028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:12.933074 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:51:12.934681 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:12.934703 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:12.935107 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:12.935475 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:12.936937 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:12.938327 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:12.939570 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:12.940617 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:12.940807 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:12.945713 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:12.945721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:12.945737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:12.945778 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:12.948799 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:12.949656 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:12.949662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:12.949672 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:12.949857 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:12.950227 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:12.950230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:12.950232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:12.950768 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:12.951050 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:12.951474 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 12:51:12.951899 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":320,"threads_started":1}
I20260924 12:51:12.960150 21750 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38449 } }
I20260924 12:51:12.960628 21750 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:12.960731 21750 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc44d7ea157b4d849e7e4c6b31b56013, State: Initialized, Role: FOLLOWER
I20260924 12:51:12.961330 21750 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38449 } }
I20260924 12:51:12.961560 21750 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:12.961687 21750 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:12.961851 21750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:12.964059 21750 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38449 } }
I20260924 12:51:12.964601 21750 leader_election.cc:302] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc44d7ea157b4d849e7e4c6b31b56013; no voters:
I20260924 12:51:12.965261 21750 leader_election.cc:288] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:12.965380 21755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:12.966482 21755 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [term 1 LEADER]: Becoming Leader. State: Replica: cc44d7ea157b4d849e7e4c6b31b56013, State: Running, Role: LEADER
I20260924 12:51:12.967195 21755 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38449 } }
I20260924 12:51:12.967499 21750 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:12.970162 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38449 } } }
I20260924 12:51:12.970230 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc44d7ea157b4d849e7e4c6b31b56013. Latest consensus state: current_term: 1 leader_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc44d7ea157b4d849e7e4c6b31b56013" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 38449 } } }
I20260924 12:51:12.970695 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:12.970695 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:12.972539 21764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:12.977324 21764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:12.986888 21764 catalog_manager.cc:1347] Generated new cluster ID: e85f77b93106455e8d2c1638e431dca0
I20260924 12:51:12.987028 21764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:12.995807 21764 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:12.997205 21764 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:13.003414 21764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013: Generated new TSK 0
I20260924 12:51:13.004304 21764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.81.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
Entry for principal kudu/127.2.81.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab.
I20260924 12:51:13.073658 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38449
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:13.444451 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:13.444747 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:13.444868 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:13.452673 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:13.452829 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:13.453100 21776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:51:13.463953 21776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.65.keytab
--principal=kudu/127.2.81.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:13.465304 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:13.466605 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:13.480254 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:13.480304 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:13.480312 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:13.480906 21776 server_base.cc:1034] running on GCE node
I20260924 12:51:13.481600 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:13.482584 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:13.483824 21776 hybrid_clock.cc:648] HybridClock initialized: now 1790254273483754 us; error 55 us; skew 500 ppm
I20260924 12:51:13.484181 21776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:13 dist-test-slave-9tm1 krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254273, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:13.487679 21776 init.cc:377] Logged in from keytab as kudu/127.2.81.65@KRBTEST.COM (short username kudu)
I20260924 12:51:13.489307 21776 webserver.cc:460] Webserver started at http://127.2.81.65:41687/ using document root <none> and password file <none>
I20260924 12:51:13.490020 21776 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:13.490140 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:13.490454 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:13.493507 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "d8adeadc0dc84409b688a97d0be93392"
format_stamp: "Formatted at 2026-09-24 12:51:13 on dist-test-slave-9tm1"
server_key: "328c4dd05c67bcc5c90ba896d4ad0215"
server_key_iv: "083dd3edeeac9592c500819d6d054c66"
server_key_version: "encryptionkey@0"
I20260924 12:51:13.494277 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "d8adeadc0dc84409b688a97d0be93392"
format_stamp: "Formatted at 2026-09-24 12:51:13 on dist-test-slave-9tm1"
server_key: "328c4dd05c67bcc5c90ba896d4ad0215"
server_key_iv: "083dd3edeeac9592c500819d6d054c66"
server_key_version: "encryptionkey@0"
I20260924 12:51:13.499689 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:13.503324 21791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:13.504717 21776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:13.504858 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "d8adeadc0dc84409b688a97d0be93392"
format_stamp: "Formatted at 2026-09-24 12:51:13 on dist-test-slave-9tm1"
server_key: "328c4dd05c67bcc5c90ba896d4ad0215"
server_key_iv: "083dd3edeeac9592c500819d6d054c66"
server_key_version: "encryptionkey@0"
I20260924 12:51:13.505097 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:13.521360 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:13.525887 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:13.526142 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:13.527125 21776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:13.528568 21776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:13.528642 21776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:13.528738 21776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:13.528788 21776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:13.560356 21776 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:36273
I20260924 12:51:13.560405 21903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:36273 every 8 connection(s)
I20260924 12:51:13.561750 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:51:13.563194 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21776
I20260924 12:51:13.563347 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:51:13.563736 2373 external_mini_cluster.cc:1428] Setting key 18a667fa764d96efe32182bcfe87283f
Sep 24 12:51:13 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254273, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.65@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:13.584196 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:13.564869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:54181 (local address 127.2.81.126:38449)
0924 12:51:13.565347 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:13.565351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:13.566601 (+ 1250us) server_negotiation.cc:407] Connection header received
0924 12:51:13.568647 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:13.568653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:13.568737 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:13.568846 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:13.570504 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:13.571529 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:13.573392 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:13.574014 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:13.574143 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:13.578275 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:13.578280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:13.578284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:13.578322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:13.580854 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:13.581685 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:13.581690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:13.581692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:13.581827 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:13.582311 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:13.582314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:13.582317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:13.582626 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:13.582784 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:13.583581 (+ 797us) server_negotiation.cc:299] Negotiation successful
0924 12:51:13.583831 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":279,"threads_started":1}
I20260924 12:51:13.585390 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:13.565541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:38449 (local address 127.2.81.65:54181)
0924 12:51:13.566426 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:13.566479 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:13.568359 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:13.569149 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:13.569158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:13.569720 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:13.570321 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:13.570340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:13.571717 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:13.571723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:13.573228 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:13.573243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:13.574264 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:13.574271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:13.574535 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:13.575262 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:13.575292 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:13.577992 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:13.581055 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:13.581065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:13.581085 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:13.581540 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:13.581963 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:13.581968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:13.581972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:13.582184 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:13.582765 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:13.582774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:13.583178 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 12:51:13.583431 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 12:51:13.584014 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":364,"threads_started":1}
I20260924 12:51:13.587205 21904 heartbeater.cc:347] Connected to a master server at 127.2.81.126:38449
I20260924 12:51:13.587507 21904 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.81.66@KRBTEST.COM; defaulting to no policy
I20260924 12:51:13.588483 21904 heartbeater.cc:511] Master 127.2.81.126:38449 requested a full tablet report, sending...
I20260924 12:51:13.590988 21715 ts_manager.cc:194] Registered new tserver with Master: d8adeadc0dc84409b688a97d0be93392 (127.2.81.65:36273)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.66@KRBTEST.COM" created.
I20260924 12:51:13.592581 21715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.65@KRBTEST.COM'} at 127.2.81.65:54181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
Entry for principal kudu/127.2.81.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab.
I20260924 12:51:13.620800 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38449
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:13.955859 21910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:13.956169 21910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:13.956265 21910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:13.963878 21910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:13.964006 21910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:13.964145 21910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:51:13.974267 21910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.66.keytab
--principal=kudu/127.2.81.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:13.975553 21910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:13.977046 21910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:13.991150 21915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:13.991111 21916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:13.991170 21918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:13.992648 21910 server_base.cc:1034] running on GCE node
I20260924 12:51:13.993368 21910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:13.994382 21910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:13.995677 21910 hybrid_clock.cc:648] HybridClock initialized: now 1790254273995619 us; error 43 us; skew 500 ppm
I20260924 12:51:13.996011 21910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:13 dist-test-slave-9tm1 krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254273, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:13.999173 21910 init.cc:377] Logged in from keytab as kudu/127.2.81.66@KRBTEST.COM (short username kudu)
I20260924 12:51:14.000691 21910 webserver.cc:460] Webserver started at http://127.2.81.66:44495/ using document root <none> and password file <none>
I20260924 12:51:14.001441 21910 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:14.001592 21910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:14.001902 21910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:14.004855 21910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "f7adc38cdf124bbab49588e537645b9c"
format_stamp: "Formatted at 2026-09-24 12:51:13 on dist-test-slave-9tm1"
server_key: "425af6d3dc5e8463061e4b3bee8d6497"
server_key_iv: "bcfa13efbe3534753fe5b3e1b122f9f3"
server_key_version: "encryptionkey@0"
I20260924 12:51:14.005730 21910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "f7adc38cdf124bbab49588e537645b9c"
format_stamp: "Formatted at 2026-09-24 12:51:13 on dist-test-slave-9tm1"
server_key: "425af6d3dc5e8463061e4b3bee8d6497"
server_key_iv: "bcfa13efbe3534753fe5b3e1b122f9f3"
server_key_version: "encryptionkey@0"
I20260924 12:51:14.011392 21910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:14.015237 21925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:14.016726 21910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:14.016903 21910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "f7adc38cdf124bbab49588e537645b9c"
format_stamp: "Formatted at 2026-09-24 12:51:13 on dist-test-slave-9tm1"
server_key: "425af6d3dc5e8463061e4b3bee8d6497"
server_key_iv: "bcfa13efbe3534753fe5b3e1b122f9f3"
server_key_version: "encryptionkey@0"
I20260924 12:51:14.017150 21910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:14.032883 21910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:14.037806 21910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:14.038097 21910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:14.039115 21910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:14.040681 21910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:14.040772 21910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:14.040892 21910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:14.040948 21910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:14.073897 21910 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:37995
I20260924 12:51:14.073961 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:37995 every 8 connection(s)
I20260924 12:51:14.075361 21910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:51:14.085412 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 21910
I20260924 12:51:14.085605 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:51:14.086148 2373 external_mini_cluster.cc:1428] Setting key 6870dcf9f674ae492c346111c4a74ebd
Sep 24 12:51:14 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254273, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.66@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:14.097323 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.078326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:53953 (local address 127.2.81.126:38449)
0924 12:51:14.078524 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.078533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.079986 (+ 1453us) server_negotiation.cc:407] Connection header received
0924 12:51:14.081950 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.081956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.082046 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.082133 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:14.083796 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.084507 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.086865 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.087421 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.087584 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.091305 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:14.091311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:14.091314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:14.091362 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:14.093941 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:14.094713 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:14.094717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:14.094720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:14.094876 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:14.095421 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:14.095427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:14.095446 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:14.095814 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:14.095980 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.096687 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.096939 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:51:14.098408 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.078838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:38449 (local address 127.2.81.66:53953)
0924 12:51:14.079764 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.079811 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.081634 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.082418 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.082429 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.082991 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:14.083595 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.083613 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.084679 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.084683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.086652 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.086671 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.087732 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.087742 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.088054 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.088781 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:14.088805 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:14.091058 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:14.094097 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:14.094108 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:14.094127 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:14.094554 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:14.095008 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:14.095013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:14.095017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:14.095286 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:14.095954 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:14.095963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:14.096326 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.096568 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.097154 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":330,"threads_started":1}
I20260924 12:51:14.099882 22038 heartbeater.cc:347] Connected to a master server at 127.2.81.126:38449
I20260924 12:51:14.100179 22038 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:14.100903 22038 heartbeater.cc:511] Master 127.2.81.126:38449 requested a full tablet report, sending...
I20260924 12:51:14.102383 21715 ts_manager.cc:194] Registered new tserver with Master: f7adc38cdf124bbab49588e537645b9c (127.2.81.66:37995)
I20260924 12:51:14.103199 21715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.66@KRBTEST.COM'} at 127.2.81.66:53953
WARNING: no policy specified for kudu/127.2.81.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
Entry for principal kudu/127.2.81.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab.
I20260924 12:51:14.139745 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38449
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:14.471519 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:14.471837 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:14.471946 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:14.479698 22043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:14.479838 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:14.479959 22043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:51:14.490083 22043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.67.keytab
--principal=kudu/127.2.81.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:38449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:14.491338 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:14.492638 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:14.505435 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:14.505499 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:14.505803 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:14.506623 22043 server_base.cc:1034] running on GCE node
I20260924 12:51:14.507238 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:14.508250 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:14.509492 22043 hybrid_clock.cc:648] HybridClock initialized: now 1790254274509446 us; error 53 us; skew 500 ppm
I20260924 12:51:14.509859 22043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:14 dist-test-slave-9tm1 krb5kdc[21670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254274, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:14.517401 22043 init.cc:377] Logged in from keytab as kudu/127.2.81.67@KRBTEST.COM (short username kudu)
I20260924 12:51:14.518980 22043 webserver.cc:460] Webserver started at http://127.2.81.67:45289/ using document root <none> and password file <none>
I20260924 12:51:14.519672 22043 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:14.519778 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:14.520066 22043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:14.523221 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "0a5b421f284b4ade9a437e19cc103ea1"
format_stamp: "Formatted at 2026-09-24 12:51:14 on dist-test-slave-9tm1"
server_key: "c1a41673a1aa03f1b87da16c4142f8cd"
server_key_iv: "f1f1472d9edb670e91486405e00e4ce3"
server_key_version: "encryptionkey@0"
I20260924 12:51:14.524094 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "0a5b421f284b4ade9a437e19cc103ea1"
format_stamp: "Formatted at 2026-09-24 12:51:14 on dist-test-slave-9tm1"
server_key: "c1a41673a1aa03f1b87da16c4142f8cd"
server_key_iv: "f1f1472d9edb670e91486405e00e4ce3"
server_key_version: "encryptionkey@0"
I20260924 12:51:14.529906 22043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:51:14.534044 22058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:14.535892 22043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:51:14.536100 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "0a5b421f284b4ade9a437e19cc103ea1"
format_stamp: "Formatted at 2026-09-24 12:51:14 on dist-test-slave-9tm1"
server_key: "c1a41673a1aa03f1b87da16c4142f8cd"
server_key_iv: "f1f1472d9edb670e91486405e00e4ce3"
server_key_version: "encryptionkey@0"
I20260924 12:51:14.536306 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:14.552920 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:14.557489 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:14.557773 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:14.558766 22043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:14.560156 22043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:14.560235 22043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:14.560348 22043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:14.560401 22043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:14.593943 22043 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:38859
I20260924 12:51:14.593986 22170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:38859 every 8 connection(s)
I20260924 12:51:14.595573 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:51:14.595851 21904 heartbeater.cc:503] Master 127.2.81.126:38449 was elected leader, sending a full tablet report...
I20260924 12:51:14.604017 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22043
I20260924 12:51:14.604207 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:51:14.604735 2373 external_mini_cluster.cc:1428] Setting key eb8e3c598b8029db92578b466b68d2e7
Sep 24 12:51:14 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254274, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.67@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:14.619133 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.599725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:39895 (local address 127.2.81.126:38449)
0924 12:51:14.600194 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.600199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.601255 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 12:51:14.603155 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.603161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.603260 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.603357 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:14.605472 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.606263 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.608521 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.609000 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.609142 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.612934 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:14.612939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:14.612942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:14.613012 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:14.615698 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:14.616531 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:14.616537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:14.616541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:14.616693 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:14.617213 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:14.617218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:14.617221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:14.617513 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:14.617664 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.618451 (+ 787us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.618722 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":249,"threads_started":1}
I20260924 12:51:14.620157 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.600263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:38449 (local address 127.2.81.67:39895)
0924 12:51:14.601089 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.601127 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.602858 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.603639 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.603651 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.604425 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:14.605275 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.605299 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.606435 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.606439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.608333 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.608352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.609248 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.609256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.609500 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.610240 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:14.610266 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:14.612655 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:14.615888 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:14.615898 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:14.615915 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:14.616396 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:14.616815 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:14.616820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:14.616824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:14.617102 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:14.617652 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:14.617660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:14.618070 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.618326 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.618890 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260924 12:51:14.621726 22171 heartbeater.cc:347] Connected to a master server at 127.2.81.126:38449
I20260924 12:51:14.622442 22171 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:14.623355 22171 heartbeater.cc:511] Master 127.2.81.126:38449 requested a full tablet report, sending...
I20260924 12:51:14.625141 21715 ts_manager.cc:194] Registered new tserver with Master: 0a5b421f284b4ade9a437e19cc103ea1 (127.2.81.67:38859)
I20260924 12:51:14.626022 21715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.81.67@KRBTEST.COM'} at 127.2.81.67:39895
I20260924 12:51:14.633802 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:51:14.650380 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.637927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.2.81.126:38449)
0924 12:51:14.638176 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.638183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.638463 (+ 280us) server_negotiation.cc:407] Connection header received
0924 12:51:14.638635 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.638641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.638716 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.638800 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:14.640209 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.640919 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.642314 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.642713 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.642834 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.645182 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:14.645186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:14.645189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:14.645225 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:14.647715 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:14.648402 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:14.648406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:14.648408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:14.648531 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:14.648887 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:14.648891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:14.648893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:14.649275 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:14.649503 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.649830 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.650059 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 12:51:14.654688 21715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:51:14.662509 21715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:51:14.688270 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.675520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55680)
0924 12:51:14.677243 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.677282 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.677467 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.678916 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.678925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.678960 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:14.679416 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.679435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.681871 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.681878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.684534 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.684564 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.686918 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.686927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.687215 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.687290 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.687435 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.687707 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"thread_start_us":2449,"threads_started":1}
I20260924 12:51:14.690307 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.675717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.2.81.65:36273)
0924 12:51:14.678206 (+ 2489us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.678215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.678241 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:51:14.678338 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.678346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.678648 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.679649 (+ 1001us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:14.679920 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.681691 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.684821 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.686637 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.689263 (+ 2626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.689607 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.689695 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.689962 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2325,"thread_start_us":591,"threads_started":1}
I20260924 12:51:14.692839 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.675939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:37995 (local address 127.0.0.1:40146)
0924 12:51:14.678224 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.678239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.678356 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.679297 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.679303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.679337 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:14.679811 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.679823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.683503 (+ 3680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.683509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.688234 (+ 4725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.688265 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.692015 (+ 3750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.692022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.692201 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.692236 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.692321 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.692473 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"thread_start_us":867,"threads_started":1}
I20260924 12:51:14.692947 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.676544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.2.81.66:37995)
0924 12:51:14.678026 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.678035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.678473 (+ 438us) server_negotiation.cc:407] Connection header received
0924 12:51:14.678586 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.678595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.678813 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.679012 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:14.681534 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.683291 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.688634 (+ 5343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.691667 (+ 3033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.691996 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.692222 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.692411 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.692600 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":1157,"threads_started":1}
I20260924 12:51:14.696349 21839 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4d586ddd4c4112b3c1151fc282777b (DEFAULT_TABLE table=test-table [id=880f22da71ab4138ad85cf15393a266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:14.697866 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4d586ddd4c4112b3c1151fc282777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:14.697803 21973 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4d586ddd4c4112b3c1151fc282777b (DEFAULT_TABLE table=test-table [id=880f22da71ab4138ad85cf15393a266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:14.698848 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.678350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:38859 (local address 127.0.0.1:40756)
0924 12:51:14.688896 (+ 10546us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.688920 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.689065 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.691792 (+ 2727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.691798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.691830 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:14.692062 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.692073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.694103 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.694107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.695684 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.695704 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.697820 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.697828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.698020 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.698058 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.698168 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.698358 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10425,"thread_start_us":11816,"threads_started":1}
I20260924 12:51:14.699092 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.678744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.2.81.67:38859)
0924 12:51:14.691129 (+ 12385us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.691138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.691171 (+ 33us) server_negotiation.cc:407] Connection header received
0924 12:51:14.691270 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.691277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.691552 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.692261 (+ 709us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:14.692537 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.693947 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.695876 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.697584 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.697775 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.698012 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.698225 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.698400 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12193,"thread_start_us":12016,"threads_started":1}
I20260924 12:51:14.700083 21973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4d586ddd4c4112b3c1151fc282777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:14.704510 22106 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4d586ddd4c4112b3c1151fc282777b (DEFAULT_TABLE table=test-table [id=880f22da71ab4138ad85cf15393a266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:14.706809 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4d586ddd4c4112b3c1151fc282777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:14.711859 22190 tablet_bootstrap.cc:492] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Bootstrap starting.
I20260924 12:51:14.715338 22191 tablet_bootstrap.cc:492] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Bootstrap starting.
I20260924 12:51:14.715631 22190 tablet_bootstrap.cc:654] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:14.717005 22190 log.cc:824] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:14.718950 22191 tablet_bootstrap.cc:654] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:14.720213 22191 log.cc:824] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:14.720386 22190 tablet_bootstrap.cc:492] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: No bootstrap required, opened a new log
I20260924 12:51:14.720676 22190 ts_tablet_manager.cc:1397] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:51:14.723317 22194 tablet_bootstrap.cc:492] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Bootstrap starting.
I20260924 12:51:14.724090 22191 tablet_bootstrap.cc:492] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: No bootstrap required, opened a new log
I20260924 12:51:14.724350 22191 ts_tablet_manager.cc:1397] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 12:51:14.727001 22194 tablet_bootstrap.cc:654] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:14.728353 22194 log.cc:824] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:14.731093 22190 raft_consensus.cc:348] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.731571 22190 raft_consensus.cc:374] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:14.731712 22190 raft_consensus.cc:729] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8adeadc0dc84409b688a97d0be93392, State: Initialized, Role: FOLLOWER
I20260924 12:51:14.731940 22194 tablet_bootstrap.cc:492] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: No bootstrap required, opened a new log
I20260924 12:51:14.731806 22191 raft_consensus.cc:348] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.732246 22191 raft_consensus.cc:374] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:14.732281 22194 ts_tablet_manager.cc:1397] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 12:51:14.732378 22191 raft_consensus.cc:729] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7adc38cdf124bbab49588e537645b9c, State: Initialized, Role: FOLLOWER
I20260924 12:51:14.732410 22190 consensus_queue.cc:260] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.733109 22191 consensus_queue.cc:260] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.735006 22038 heartbeater.cc:503] Master 127.2.81.126:38449 was elected leader, sending a full tablet report...
I20260924 12:51:14.736168 22190 ts_tablet_manager.cc:1428] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260924 12:51:14.737455 22191 ts_tablet_manager.cc:1428] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260924 12:51:14.742519 22194 raft_consensus.cc:348] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.742924 22194 raft_consensus.cc:374] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:14.743024 22194 raft_consensus.cc:729] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5b421f284b4ade9a437e19cc103ea1, State: Initialized, Role: FOLLOWER
I20260924 12:51:14.743602 22194 consensus_queue.cc:260] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.744639 22171 heartbeater.cc:503] Master 127.2.81.126:38449 was elected leader, sending a full tablet report...
I20260924 12:51:14.745324 22194 ts_tablet_manager.cc:1428] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260924 12:51:14.814570 21905 tablet.cc:2369] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:51:14.827487 22039 tablet.cc:2369] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 12:51:14.848269 22172 tablet.cc:2369] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:14.990788 22197 raft_consensus.cc:482] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:51:14.991192 22197 raft_consensus.cc:504] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:14.992944 22197 leader_election.cc:288] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8adeadc0dc84409b688a97d0be93392 (127.2.81.65:36273), 0a5b421f284b4ade9a437e19cc103ea1 (127.2.81.67:38859)
I20260924 12:51:14.999215 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.993325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:43059 (local address 127.2.81.65:36273)
0924 12:51:14.993587 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.993594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.993910 (+ 316us) server_negotiation.cc:407] Connection header received
0924 12:51:14.994116 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.994122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.994205 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.994300 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:14.994843 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.995559 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.997487 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.998285 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.998414 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.998521 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.998834 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.998964 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 12:51:14.999295 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.993192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.2.81.66:43059)
0924 12:51:14.993792 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.993808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.993968 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.994345 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.994351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.994381 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:14.994694 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.994703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.995761 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.995767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.997312 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.997327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.998527 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.998534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.998674 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.998699 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.998783 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.998933 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260924 12:51:14.999527 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.994017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:59241 (local address 127.2.81.67:38859)
0924 12:51:14.994212 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:14.994217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:14.994590 (+ 373us) server_negotiation.cc:407] Connection header received
0924 12:51:14.994753 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:14.994758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:14.994820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:14.994895 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:14.995388 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.996132 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.997858 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.998639 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.998755 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.998853 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:14.999161 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 12:51:14.999289 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:51:14.999754 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:14.993901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:38859 (local address 127.2.81.66:59241)
0924 12:51:14.994492 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:14.994508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:14.994627 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:14.994922 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:14.994927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:14.994949 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:14.995249 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.995258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.996280 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.996285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.997701 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:14.997713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:14.998854 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:14.998860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:14.999016 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:14.999039 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 12:51:14.999121 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 12:51:14.999250 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260924 12:51:15.000385 21859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4d586ddd4c4112b3c1151fc282777b" candidate_uuid: "f7adc38cdf124bbab49588e537645b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8adeadc0dc84409b688a97d0be93392" is_pre_election: true
I20260924 12:51:15.000651 22126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4d586ddd4c4112b3c1151fc282777b" candidate_uuid: "f7adc38cdf124bbab49588e537645b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a5b421f284b4ade9a437e19cc103ea1" is_pre_election: true
I20260924 12:51:15.001031 21859 raft_consensus.cc:2454] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7adc38cdf124bbab49588e537645b9c in term 0.
I20260924 12:51:15.001149 22126 raft_consensus.cc:2454] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7adc38cdf124bbab49588e537645b9c in term 0.
I20260924 12:51:15.001794 21927 leader_election.cc:302] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5b421f284b4ade9a437e19cc103ea1, f7adc38cdf124bbab49588e537645b9c; no voters:
I20260924 12:51:15.002166 22197 raft_consensus.cc:2781] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:51:15.002313 22197 raft_consensus.cc:482] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:51:15.002426 22197 raft_consensus.cc:3037] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:15.004626 22197 raft_consensus.cc:504] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:15.005589 22197 leader_election.cc:288] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [CANDIDATE]: Term 1 election: Requested vote from peers d8adeadc0dc84409b688a97d0be93392 (127.2.81.65:36273), 0a5b421f284b4ade9a437e19cc103ea1 (127.2.81.67:38859)
I20260924 12:51:15.006254 22126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4d586ddd4c4112b3c1151fc282777b" candidate_uuid: "f7adc38cdf124bbab49588e537645b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a5b421f284b4ade9a437e19cc103ea1"
I20260924 12:51:15.006255 21859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4d586ddd4c4112b3c1151fc282777b" candidate_uuid: "f7adc38cdf124bbab49588e537645b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8adeadc0dc84409b688a97d0be93392"
I20260924 12:51:15.006526 22126 raft_consensus.cc:3037] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:15.006525 21859 raft_consensus.cc:3037] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:15.009802 22126 raft_consensus.cc:2454] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7adc38cdf124bbab49588e537645b9c in term 1.
I20260924 12:51:15.009807 21859 raft_consensus.cc:2454] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7adc38cdf124bbab49588e537645b9c in term 1.
I20260924 12:51:15.010275 21927 leader_election.cc:302] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8adeadc0dc84409b688a97d0be93392, f7adc38cdf124bbab49588e537645b9c; no voters:
I20260924 12:51:15.010584 22197 raft_consensus.cc:2781] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:15.011049 22197 raft_consensus.cc:686] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 1 LEADER]: Becoming Leader. State: Replica: f7adc38cdf124bbab49588e537645b9c, State: Running, Role: LEADER
I20260924 12:51:15.011523 22197 consensus_queue.cc:237] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } }
I20260924 12:51:15.016191 21714 catalog_manager.cc:5495] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c reported cstate change: term changed from 0 to 1, leader changed from <none> to f7adc38cdf124bbab49588e537645b9c (127.2.81.66). New cstate: current_term: 1 leader_uuid: "f7adc38cdf124bbab49588e537645b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7adc38cdf124bbab49588e537645b9c" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 37995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 } health_report { overall_health: UNKNOWN } } }
I20260924 12:51:15.063236 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.057245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.2.81.66:37995)
0924 12:51:15.057465 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:15.057472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:15.057496 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:51:15.057738 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:15.057743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:15.057828 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:15.057966 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:15.058385 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.059291 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.060791 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.061278 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.061385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:15.061975 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:15.062529 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:15.062678 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:15.062862 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 12:51:15.062949 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 12:51:15.072194 22126 raft_consensus.cc:1261] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 1 FOLLOWER]: Refusing update from remote peer f7adc38cdf124bbab49588e537645b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:15.072194 21859 raft_consensus.cc:1261] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 1 FOLLOWER]: Refusing update from remote peer f7adc38cdf124bbab49588e537645b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:15.073731 22201 consensus_queue.cc:1035] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8adeadc0dc84409b688a97d0be93392" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:15.074270 22197 consensus_queue.cc:1035] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a5b421f284b4ade9a437e19cc103ea1" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:15.090162 22204 mvcc.cc:205] Tried to move back new op lower bound from 7332881510684839936 to 7332881510450561024. Current Snapshot: MvccSnapshot[applied={T|T < 7332881510684839936}]
I20260924 12:51:15.107013 21714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:15.107345 21714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:15.111419 21714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc44d7ea157b4d849e7e4c6b31b56013: Sending DeleteTablet for 3 replicas of tablet 9d4d586ddd4c4112b3c1151fc282777b
I20260924 12:51:15.112730 21839 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4d586ddd4c4112b3c1151fc282777b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:15 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:55680
I20260924 12:51:15.113209 22106 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4d586ddd4c4112b3c1151fc282777b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:15 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:40756
I20260924 12:51:15.112740 21973 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4d586ddd4c4112b3c1151fc282777b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:15 UTC) from {username='kudu', principal='kudu/127.2.81.126@KRBTEST.COM'} at 127.0.0.1:40146
I20260924 12:51:15.114127 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21776
I20260924 12:51:15.136020 22215 tablet_replica.cc:330] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: stopping tablet replica
I20260924 12:51:15.137009 22215 raft_consensus.cc:2229] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:51:15.137509 22214 tablet_replica.cc:330] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: stopping tablet replica
I20260924 12:51:15.137980 22214 raft_consensus.cc:2229] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:15.138738 22215 raft_consensus.cc:2258] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:15.139313 22214 raft_consensus.cc:2258] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:15.140329 22213 tablet_replica.cc:330] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: stopping tablet replica
I20260924 12:51:15.142338 22214 ts_tablet_manager.cc:1905] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:15.142445 22215 ts_tablet_manager.cc:1905] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:15.143039 22213 raft_consensus.cc:2229] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:15.143646 22213 raft_consensus.cc:2258] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 12:51:15 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.65@KRBTEST.COM
I20260924 12:51:15.149498 22213 ts_tablet_manager.cc:1905] T 9d4d586ddd4c4112b3c1151fc282777b P d8adeadc0dc84409b688a97d0be93392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:15.151527 22214 ts_tablet_manager.cc:1918] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:15.151777 22214 log.cc:1196] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/9d4d586ddd4c4112b3c1151fc282777b
I20260924 12:51:15.151742 22215 ts_tablet_manager.cc:1918] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:15.152024 22215 log.cc:1196] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/9d4d586ddd4c4112b3c1151fc282777b
I20260924 12:51:15.152359 22214 ts_tablet_manager.cc:1939] T 9d4d586ddd4c4112b3c1151fc282777b P 0a5b421f284b4ade9a437e19cc103ea1: Deleting consensus metadata
I20260924 12:51:15.152690 22215 ts_tablet_manager.cc:1939] T 9d4d586ddd4c4112b3c1151fc282777b P f7adc38cdf124bbab49588e537645b9c: Deleting consensus metadata
I20260924 12:51:15.154752 21700 catalog_manager.cc:4841] TS 0a5b421f284b4ade9a437e19cc103ea1 (127.2.81.67:38859): tablet 9d4d586ddd4c4112b3c1151fc282777b (table test-table [id=880f22da71ab4138ad85cf15393a266c]) successfully deleted
I20260924 12:51:15.155135 21701 catalog_manager.cc:4841] TS f7adc38cdf124bbab49588e537645b9c (127.2.81.66:37995): tablet 9d4d586ddd4c4112b3c1151fc282777b (table test-table [id=880f22da71ab4138ad85cf15393a266c]) successfully deleted
I20260924 12:51:15.159119 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.115579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.2.81.65:36273)
0924 12:51:15.115763 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:15.115771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:15.135686 (+ 19915us) server_negotiation.cc:407] Connection header received
0924 12:51:15.136186 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:15.136194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:15.136297 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:15.136375 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:15.138514 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.139885 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.142267 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.142981 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.143112 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:15.152212 (+ 9100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:15.152235 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:15.152257 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:15.152312 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:15.156069 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:15.156872 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:15.156880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:15.156883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:15.157076 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:15.157452 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:15.157456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:15.157458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:15.157840 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:15.158126 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:15.158445 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 12:51:15.158704 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:51:15.159848 21899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:15.298206 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21776
W20260924 12:51:15.312386 21700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:36273 (error 108)
I20260924 12:51:15.313177 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21910
I20260924 12:51:15.314687 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.313902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55700)
0924 12:51:15.314373 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.314500 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
W20260924 12:51:15.315110 21700 catalog_manager.cc:4584] TS d8adeadc0dc84409b688a97d0be93392 (127.2.81.65:36273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d4d586ddd4c4112b3c1151fc282777b: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Sep 24 12:51:15 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.66@KRBTEST.COM
I20260924 12:51:15.330742 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.313970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.2.81.66:37995)
0924 12:51:15.314454 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:15.314461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:15.314483 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:51:15.314548 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:15.314554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:15.314652 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:15.314757 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:15.316071 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.317378 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.318758 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.319461 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.319620 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:15.324367 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:15.324382 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:15.324399 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:15.324457 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:15.327480 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:15.328453 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:15.328464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:15.328468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:15.328660 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:15.329151 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:15.329156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:15.329158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:15.329573 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:15.329865 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:15.330207 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 12:51:15.330419 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344}
I20260924 12:51:15.331423 22033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:15.368850 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.368312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55702)
0924 12:51:15.368556 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.368671 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 12:51:15.370007 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.369567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55716)
0924 12:51:15.369723 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.369828 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 12:51:15.430764 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.430232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55718)
0924 12:51:15.430472 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.430592 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 12:51:15.431807 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.431395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55720)
0924 12:51:15.431554 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.431635 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 12:51:15.446583 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21910
I20260924 12:51:15.462182 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22043
Sep 24 12:51:15 dist-test-slave-9tm1 krb5kdc[21670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.67@KRBTEST.COM
I20260924 12:51:15.478204 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.462993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.2.81.67:38859)
0924 12:51:15.463198 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:15.463204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:15.463225 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:51:15.463379 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:15.463383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:15.463449 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:15.463542 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:15.465205 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.465936 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.467285 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:15.467677 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:15.467789 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:15.472030 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:15.472046 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:15.472063 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:15.472103 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:15.475068 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:15.475872 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:15.475880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:15.475882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:15.476046 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:15.476432 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:15.476436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:15.476438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:15.476824 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:15.477161 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:15.477450 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 12:51:15.477706 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 12:51:15.478899 22166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:15.512539 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.511985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55724)
0924 12:51:15.512240 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.512356 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 12:51:15.513738 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.513252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55738)
0924 12:51:15.513469 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.513570 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 12:51:15.572047 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22043
I20260924 12:51:15.584911 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 21683
I20260924 12:51:15.586148 21745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:15.700222 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:15.699657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36273 (local address 127.0.0.1:55744)
0924 12:51:15.699877 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:15.700013 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 12:51:15.700450 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 21683
2026-09-24T12:51:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4516 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T12:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:15Z Disabled control of system clock
I20260924 12:51:15.772827 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37935
--webserver_interface=127.2.81.126
--webserver_port=0
--builtin_ntp_servers=127.2.81.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 12:51:16.106169 22226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:16.106457 22226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:16.106557 22226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:16.113981 22226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:16.114097 22226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:16.114142 22226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:16.114224 22226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:16.114276 22226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:16.123800 22226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:37935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:16.125136 22226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:16.126976 22226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:16.138696 22234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:16.138652 22232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:16.138653 22231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:16.139499 22226 server_base.cc:1034] running on GCE node
I20260924 12:51:16.140261 22226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:16.141762 22226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:16.142979 22226 hybrid_clock.cc:648] HybridClock initialized: now 1790254276142930 us; error 29 us; skew 500 ppm
I20260924 12:51:16.143309 22226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:16.145649 22226 webserver.cc:460] Webserver started at http://127.2.81.126:42675/ using document root <none> and password file <none>
I20260924 12:51:16.146371 22226 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:16.146468 22226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:16.146754 22226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:16.149714 22226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "29a6025fa39f4213ba9a3de39dcb3216"
format_stamp: "Formatted at 2026-09-24 12:51:16 on dist-test-slave-9tm1"
server_key: "adaf55c2506fd555057d8f92afe5062d"
server_key_iv: "42eadcdc5b8b39881b80f3531a399c06"
server_key_version: "encryptionkey@0"
I20260924 12:51:16.150463 22226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "29a6025fa39f4213ba9a3de39dcb3216"
format_stamp: "Formatted at 2026-09-24 12:51:16 on dist-test-slave-9tm1"
server_key: "adaf55c2506fd555057d8f92afe5062d"
server_key_iv: "42eadcdc5b8b39881b80f3531a399c06"
server_key_version: "encryptionkey@0"
I20260924 12:51:16.155618 22226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:16.159323 22240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:16.160837 22226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:16.161054 22226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "29a6025fa39f4213ba9a3de39dcb3216"
format_stamp: "Formatted at 2026-09-24 12:51:16 on dist-test-slave-9tm1"
server_key: "adaf55c2506fd555057d8f92afe5062d"
server_key_iv: "42eadcdc5b8b39881b80f3531a399c06"
server_key_version: "encryptionkey@0"
I20260924 12:51:16.161284 22226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:16.188004 22226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:16.188943 22226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:16.189251 22226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:16.207146 22226 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:37935
I20260924 12:51:16.207172 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:37935 every 8 connection(s)
I20260924 12:51:16.208563 22226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:16.212354 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:16.217722 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22226
I20260924 12:51:16.217892 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:16.218333 2373 external_mini_cluster.cc:1428] Setting key 87857fe87a45ff7f2f57a5b885cf2c07
I20260924 12:51:16.223415 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216: Bootstrap starting.
I20260924 12:51:16.226914 22292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:16.227986 22292 log.cc:824] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:16.231380 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216: No bootstrap required, opened a new log
I20260924 12:51:16.231972 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:16.220200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.2.81.126:37935)
0924 12:51:16.221244 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:16.221258 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:16.221304 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:51:16.223077 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:16.223101 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:16.223608 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:16.223997 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:16.224572 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:16.225989 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:16.227376 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:16.228099 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:16.228262 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:16.229553 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:16.229562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:16.229578 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:16.230158 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:16.230240 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:16.230266 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:16.230364 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:16.230649 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 12:51:16.230941 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":320,"threads_started":1}
I20260924 12:51:16.239776 22292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a6025fa39f4213ba9a3de39dcb3216" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37935 } }
I20260924 12:51:16.240247 22292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:16.240355 22292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a6025fa39f4213ba9a3de39dcb3216, State: Initialized, Role: FOLLOWER
I20260924 12:51:16.240933 22292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a6025fa39f4213ba9a3de39dcb3216" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37935 } }
I20260924 12:51:16.241232 22292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:16.241353 22292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:16.241510 22292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:16.243665 22292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a6025fa39f4213ba9a3de39dcb3216" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37935 } }
I20260924 12:51:16.244191 22292 leader_election.cc:302] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29a6025fa39f4213ba9a3de39dcb3216; no voters:
I20260924 12:51:16.244804 22292 leader_election.cc:288] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:16.245002 22297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:16.245673 22297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [term 1 LEADER]: Becoming Leader. State: Replica: 29a6025fa39f4213ba9a3de39dcb3216, State: Running, Role: LEADER
I20260924 12:51:16.246271 22297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a6025fa39f4213ba9a3de39dcb3216" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37935 } }
I20260924 12:51:16.246907 22292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:16.249857 22298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29a6025fa39f4213ba9a3de39dcb3216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a6025fa39f4213ba9a3de39dcb3216" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37935 } } }
I20260924 12:51:16.250347 22298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:16.249681 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29a6025fa39f4213ba9a3de39dcb3216. Latest consensus state: current_term: 1 leader_uuid: "29a6025fa39f4213ba9a3de39dcb3216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a6025fa39f4213ba9a3de39dcb3216" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 37935 } } }
I20260924 12:51:16.252734 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:16.254870 22306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:16.259404 22306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:16.268841 22306 catalog_manager.cc:1347] Generated new cluster ID: c27161c83b5c4a728fc89550e380f829
I20260924 12:51:16.269001 22306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:16.296479 22306 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:16.297899 22306 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:16.304442 22306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216: Generated new TSK 0
I20260924 12:51:16.305598 22306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:16.318723 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.65:0
--local_ip_for_outbound_sockets=127.2.81.65
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37935
--builtin_ntp_servers=127.2.81.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 12:51:16.656363 22316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:16.656662 22316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:16.656771 22316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:16.664561 22316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:16.664693 22316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:16.664853 22316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.65
I20260924 12:51:16.674597 22316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.81.65
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.65
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:16.676146 22316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:16.678087 22316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:16.691962 22321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:16.692000 22322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:16.692027 22324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:16.692956 22316 server_base.cc:1034] running on GCE node
I20260924 12:51:16.693717 22316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:16.694747 22316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:16.695976 22316 hybrid_clock.cc:648] HybridClock initialized: now 1790254276695903 us; error 68 us; skew 500 ppm
I20260924 12:51:16.696349 22316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:16.698930 22316 webserver.cc:460] Webserver started at http://127.2.81.65:45581/ using document root <none> and password file <none>
I20260924 12:51:16.699663 22316 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:16.699774 22316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:16.700225 22316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:16.703619 22316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data/instance:
uuid: "d31aee853ad84f8783a221d10be550e4"
format_stamp: "Formatted at 2026-09-24 12:51:16 on dist-test-slave-9tm1"
server_key: "8ebe45711c61d5a85633f565564a4298"
server_key_iv: "625eefe6665e652f21569d783b03f1a4"
server_key_version: "encryptionkey@0"
I20260924 12:51:16.704519 22316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance:
uuid: "d31aee853ad84f8783a221d10be550e4"
format_stamp: "Formatted at 2026-09-24 12:51:16 on dist-test-slave-9tm1"
server_key: "8ebe45711c61d5a85633f565564a4298"
server_key_iv: "625eefe6665e652f21569d783b03f1a4"
server_key_version: "encryptionkey@0"
I20260924 12:51:16.710145 22316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 12:51:16.713922 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:16.715368 22316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:16.715551 22316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal
uuid: "d31aee853ad84f8783a221d10be550e4"
format_stamp: "Formatted at 2026-09-24 12:51:16 on dist-test-slave-9tm1"
server_key: "8ebe45711c61d5a85633f565564a4298"
server_key_iv: "625eefe6665e652f21569d783b03f1a4"
server_key_version: "encryptionkey@0"
I20260924 12:51:16.715777 22316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:16.742367 22316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:16.743259 22316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:16.743531 22316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:16.744558 22316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:16.746138 22316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:16.746236 22316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:16.746358 22316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:16.746431 22316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:16.780017 22316 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.65:36311
I20260924 12:51:16.780051 22442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.65:36311 every 8 connection(s)
I20260924 12:51:16.781373 22316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/data/info.pb
I20260924 12:51:16.782878 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22316
I20260924 12:51:16.783052 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-0/wal/instance
I20260924 12:51:16.783479 2373 external_mini_cluster.cc:1428] Setting key a4946f5b364bff827c19df4f7c6068b2
I20260924 12:51:16.786623 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.66:0
--local_ip_for_outbound_sockets=127.2.81.66
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37935
--builtin_ntp_servers=127.2.81.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 12:51:16.797199 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:16.785616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.65:54201 (local address 127.2.81.126:37935)
0924 12:51:16.786091 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:16.786098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:16.787520 (+ 1422us) server_negotiation.cc:407] Connection header received
0924 12:51:16.789584 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:16.789589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:16.789681 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:16.789847 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:16.791029 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:16.792009 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:16.793943 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:16.794394 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:16.794535 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:16.796245 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:16.796251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:16.796254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:16.796277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:16.796326 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:16.796339 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:16.796420 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:16.796676 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 12:51:16.796797 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260924 12:51:16.798331 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:16.786261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37935 (local address 127.2.81.65:54201)
0924 12:51:16.787343 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:16.787387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:16.789278 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:16.790048 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:16.790057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:16.790172 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:16.790829 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:16.790850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:16.792216 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:16.792220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:16.793745 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:16.793764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:16.794673 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:16.794681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:16.794949 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:16.795765 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:16.795768 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:16.795820 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:16.795921 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:16.796442 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:16.796455 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:16.796462 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 12:51:16.796640 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 12:51:16.797174 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":363,"threads_started":1}
I20260924 12:51:16.800102 22443 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37935
I20260924 12:51:16.800479 22443 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:16.801553 22443 heartbeater.cc:511] Master 127.2.81.126:37935 requested a full tablet report, sending...
I20260924 12:51:16.804327 22257 ts_manager.cc:194] Registered new tserver with Master: d31aee853ad84f8783a221d10be550e4 (127.2.81.65:36311)
I20260924 12:51:16.806492 22257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.65:54201
W20260924 12:51:17.123101 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:17.123376 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:17.123461 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:17.130745 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:17.130858 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:17.130972 22447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.66
I20260924 12:51:17.140231 22447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.81.66
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.66
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:17.141526 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:17.143260 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:17.156116 22452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:17.156020 22453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:17.156049 22455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:17.157198 22447 server_base.cc:1034] running on GCE node
I20260924 12:51:17.157913 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:17.158999 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:17.160233 22447 hybrid_clock.cc:648] HybridClock initialized: now 1790254277160166 us; error 63 us; skew 500 ppm
I20260924 12:51:17.160552 22447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:17.163008 22447 webserver.cc:460] Webserver started at http://127.2.81.66:37017/ using document root <none> and password file <none>
I20260924 12:51:17.163694 22447 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:17.163796 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:17.164108 22447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:17.167254 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data/instance:
uuid: "9debeba37ccc4264bc8a6ca0d2ff6306"
format_stamp: "Formatted at 2026-09-24 12:51:17 on dist-test-slave-9tm1"
server_key: "229ddf7b3dde060f5a4b8313c8ab4c8f"
server_key_iv: "235e958aab3400d37f8da609cca58253"
server_key_version: "encryptionkey@0"
I20260924 12:51:17.168084 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance:
uuid: "9debeba37ccc4264bc8a6ca0d2ff6306"
format_stamp: "Formatted at 2026-09-24 12:51:17 on dist-test-slave-9tm1"
server_key: "229ddf7b3dde060f5a4b8313c8ab4c8f"
server_key_iv: "235e958aab3400d37f8da609cca58253"
server_key_version: "encryptionkey@0"
I20260924 12:51:17.173633 22447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:51:17.177788 22461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:17.179411 22447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:17.179610 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal
uuid: "9debeba37ccc4264bc8a6ca0d2ff6306"
format_stamp: "Formatted at 2026-09-24 12:51:17 on dist-test-slave-9tm1"
server_key: "229ddf7b3dde060f5a4b8313c8ab4c8f"
server_key_iv: "235e958aab3400d37f8da609cca58253"
server_key_version: "encryptionkey@0"
I20260924 12:51:17.179838 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:17.191921 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:17.192821 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:17.193186 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:17.194432 22447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:17.196074 22447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:17.196178 22447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:17.196296 22447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:17.196372 22447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:17.235347 22447 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.66:35863
I20260924 12:51:17.235383 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.66:35863 every 8 connection(s)
I20260924 12:51:17.236608 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/data/info.pb
I20260924 12:51:17.244587 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22447
I20260924 12:51:17.244769 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal/instance
I20260924 12:51:17.245365 2373 external_mini_cluster.cc:1428] Setting key 08b7f55117f42c257061a939e28166a5
I20260924 12:51:17.248687 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.81.67:0
--local_ip_for_outbound_sockets=127.2.81.67
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37935
--builtin_ntp_servers=127.2.81.84:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 12:51:17.249876 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.239127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.66:40157 (local address 127.2.81.126:37935)
0924 12:51:17.239308 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.239312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.240717 (+ 1405us) server_negotiation.cc:407] Connection header received
0924 12:51:17.242498 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.242503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.242577 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.242667 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:17.243732 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.244463 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.246677 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.247311 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.247450 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.249095 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:17.249100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:17.249103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:17.249122 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:17.249155 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:17.249167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:17.249237 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.249538 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.249629 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:51:17.251202 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.239606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37935 (local address 127.2.81.66:40157)
0924 12:51:17.240548 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.240588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.242210 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.242860 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.242869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.242989 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:17.243560 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.243578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.244635 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.244641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.246523 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.246536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.247593 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.247601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.247898 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.248696 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:17.248700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:17.248767 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:17.248879 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:17.249255 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:17.249267 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:17.249275 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.249466 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.249897 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":331,"threads_started":1}
I20260924 12:51:17.252609 22574 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37935
I20260924 12:51:17.252918 22574 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:17.253860 22574 heartbeater.cc:511] Master 127.2.81.126:37935 requested a full tablet report, sending...
I20260924 12:51:17.255477 22257 ts_manager.cc:194] Registered new tserver with Master: 9debeba37ccc4264bc8a6ca0d2ff6306 (127.2.81.66:35863)
I20260924 12:51:17.256302 22257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.66:40157
W20260924 12:51:17.577950 22577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:17.578248 22577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:17.578337 22577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:17.585778 22577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:17.585892 22577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:17.585997 22577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.81.67
I20260924 12:51:17.595366 22577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.81.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.81.67
--webserver_port=0
--tserver_master_addrs=127.2.81.126:37935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.81.67
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:17.596618 22577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:17.598325 22577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:17.611361 22582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:17.611421 22583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:17.611387 22585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:17.612267 22577 server_base.cc:1034] running on GCE node
I20260924 12:51:17.612884 22577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:17.614003 22577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:17.615224 22577 hybrid_clock.cc:648] HybridClock initialized: now 1790254277615175 us; error 43 us; skew 500 ppm
I20260924 12:51:17.615518 22577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 12:51:17.618213 22577 webserver.cc:460] Webserver started at http://127.2.81.67:44661/ using document root <none> and password file <none>
I20260924 12:51:17.618999 22577 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:17.619094 22577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:17.619408 22577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:17.622478 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data/instance:
uuid: "86b5d3495da0423db03188f0995528b8"
format_stamp: "Formatted at 2026-09-24 12:51:17 on dist-test-slave-9tm1"
server_key: "84210d9833386aa0c32a40a71a45769a"
server_key_iv: "192dcaeac9f56b57ccf0d1a56bb52c95"
server_key_version: "encryptionkey@0"
I20260924 12:51:17.623310 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance:
uuid: "86b5d3495da0423db03188f0995528b8"
format_stamp: "Formatted at 2026-09-24 12:51:17 on dist-test-slave-9tm1"
server_key: "84210d9833386aa0c32a40a71a45769a"
server_key_iv: "192dcaeac9f56b57ccf0d1a56bb52c95"
server_key_version: "encryptionkey@0"
I20260924 12:51:17.628809 22577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:51:17.633157 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:17.634799 22577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:17.634974 22577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal
uuid: "86b5d3495da0423db03188f0995528b8"
format_stamp: "Formatted at 2026-09-24 12:51:17 on dist-test-slave-9tm1"
server_key: "84210d9833386aa0c32a40a71a45769a"
server_key_iv: "192dcaeac9f56b57ccf0d1a56bb52c95"
server_key_version: "encryptionkey@0"
I20260924 12:51:17.635201 22577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:17.688356 22577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:17.689266 22577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:17.689514 22577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:17.690517 22577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 12:51:17.692153 22577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 12:51:17.692250 22577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:17.692328 22577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 12:51:17.692382 22577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:17.724180 22577 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.67:43503
I20260924 12:51:17.724232 22703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.67:43503 every 8 connection(s)
I20260924 12:51:17.725559 22577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/data/info.pb
I20260924 12:51:17.735565 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22577
I20260924 12:51:17.735728 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal/instance
I20260924 12:51:17.736174 2373 external_mini_cluster.cc:1428] Setting key ae0b27b21912408ae9006a8d306f5cb0
I20260924 12:51:17.740289 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.728593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:58779 (local address 127.2.81.126:37935)
0924 12:51:17.728784 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.728789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.730343 (+ 1554us) server_negotiation.cc:407] Connection header received
0924 12:51:17.732169 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.732176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.732267 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.732358 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:17.733829 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.734588 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.736400 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.736966 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.737149 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.739272 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:17.739278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:17.739282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:17.739302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:17.739338 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:17.739352 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:17.739447 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.739913 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.740032 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:51:17.741444 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.729166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:37935 (local address 127.2.81.67:58779)
0924 12:51:17.730175 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.730211 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.731870 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.732722 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.732732 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.732837 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:17.733638 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.733658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.734752 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.734756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.736250 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.736268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.737216 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.737223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.737556 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.738788 (+ 1232us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 12:51:17.738796 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:17.738885 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 12:51:17.739011 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:17.739497 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:17.739509 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:17.739518 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.739831 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.740254 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":398,"threads_started":1}
I20260924 12:51:17.742889 22704 heartbeater.cc:347] Connected to a master server at 127.2.81.126:37935
I20260924 12:51:17.743222 22704 heartbeater.cc:464] Registering TS with master...
I20260924 12:51:17.744048 22704 heartbeater.cc:511] Master 127.2.81.126:37935 requested a full tablet report, sending...
I20260924 12:51:17.745683 22257 ts_manager.cc:194] Registered new tserver with Master: 86b5d3495da0423db03188f0995528b8 (127.2.81.67:43503)
I20260924 12:51:17.746538 22257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.81.67:58779
I20260924 12:51:17.754278 2373 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 12:51:17.765792 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.759085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.2.81.126:37935)
0924 12:51:17.759260 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.759264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.759581 (+ 317us) server_negotiation.cc:407] Connection header received
0924 12:51:17.759669 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.759673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.759828 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.759950 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:17.760449 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.761249 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.762950 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.763439 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.763579 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.764847 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:17.764852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:17.764855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:17.764877 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:17.764920 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:17.764934 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:17.765048 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.765360 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.765475 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 12:51:17.771495 22257 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 12:51:17.779632 22257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 12:51:17.810264 22443 heartbeater.cc:503] Master 127.2.81.126:37935 was elected leader, sending a full tablet report...
I20260924 12:51:17.818229 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.794865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.2.81.65:36311)
0924 12:51:17.809746 (+ 14881us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.809754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.809777 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:51:17.809877 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.809885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.810164 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.811314 (+ 1150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:17.811522 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.812768 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.815496 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.817016 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.817152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.817442 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.817669 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.817852 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14712,"thread_start_us":14599,"threads_started":1}
I20260924 12:51:17.818279 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.794570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55336)
0924 12:51:17.807670 (+ 13100us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.807725 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.807949 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.810589 (+ 2640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.810596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.810647 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:17.811074 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.811089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.812904 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.812909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.815256 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.815273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.817201 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.817208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.817405 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.817458 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.817611 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.817825 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12757,"spinlock_wait_cycles":1408,"thread_start_us":15094,"threads_started":1}
I20260924 12:51:17.819205 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.798569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.67:43503 (local address 127.0.0.1:40884)
0924 12:51:17.810499 (+ 11930us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.810516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.810663 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.811590 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.811595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.811618 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:17.812015 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.812039 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.813927 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.813932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.815978 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.815994 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.818130 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.818136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.818503 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.818535 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.818625 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.818756 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11821,"mutex_wait_us":135,"spinlock_wait_cycles":7680,"thread_start_us":13420,"threads_started":1}
I20260924 12:51:17.819363 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.795584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.2.81.66:35863)
0924 12:51:17.809317 (+ 13733us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.809326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.809646 (+ 320us) server_negotiation.cc:407] Connection header received
0924 12:51:17.810126 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.810134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.810329 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.810481 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:17.812868 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.813915 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.816330 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.817732 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.817858 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.818085 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.818802 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.819032 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13566,"thread_start_us":13427,"threads_started":1}
I20260924 12:51:17.819229 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.795234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35863 (local address 127.0.0.1:54182)
0924 12:51:17.809523 (+ 14289us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.809541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.809919 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.812288 (+ 2369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.812292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.812316 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:17.812668 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.812681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.814122 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.814127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.816112 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.816128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.818270 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.818278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.818584 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.818616 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.818696 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.818822 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14143,"mutex_wait_us":38,"thread_start_us":12940,"threads_started":1}
I20260924 12:51:17.820310 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.799264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.2.81.67:43503)
0924 12:51:17.810133 (+ 10869us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.810141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.810805 (+ 664us) server_negotiation.cc:407] Connection header received
0924 12:51:17.810912 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.810919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.811135 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.811325 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:17.812238 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.813778 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.816213 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.817632 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.817768 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.818001 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.819769 (+ 1768us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.819971 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10708,"thread_start_us":10572,"threads_started":1}
I20260924 12:51:17.822860 22378 tablet_service.cc:1467] Processing CreateTablet for tablet 4d395cfd3d2b428f94d60e13b7d23ed8 (DEFAULT_TABLE table=test-table [id=0ab47b0c06224c2e81279c5eca65c1a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:17.823161 22509 tablet_service.cc:1467] Processing CreateTablet for tablet 4d395cfd3d2b428f94d60e13b7d23ed8 (DEFAULT_TABLE table=test-table [id=0ab47b0c06224c2e81279c5eca65c1a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:17.824271 22378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d395cfd3d2b428f94d60e13b7d23ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:17.823843 22639 tablet_service.cc:1467] Processing CreateTablet for tablet 4d395cfd3d2b428f94d60e13b7d23ed8 (DEFAULT_TABLE table=test-table [id=0ab47b0c06224c2e81279c5eca65c1a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 12:51:17.824517 22509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d395cfd3d2b428f94d60e13b7d23ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:17.825237 22639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d395cfd3d2b428f94d60e13b7d23ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:17.835740 22722 tablet_bootstrap.cc:492] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: Bootstrap starting.
I20260924 12:51:17.839907 22724 tablet_bootstrap.cc:492] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Bootstrap starting.
I20260924 12:51:17.840732 22722 tablet_bootstrap.cc:654] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:17.841611 22723 tablet_bootstrap.cc:492] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Bootstrap starting.
I20260924 12:51:17.842170 22722 log.cc:824] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:17.843533 22724 tablet_bootstrap.cc:654] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:17.844794 22724 log.cc:824] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:17.845207 22723 tablet_bootstrap.cc:654] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:17.845742 22722 tablet_bootstrap.cc:492] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: No bootstrap required, opened a new log
I20260924 12:51:17.846005 22722 ts_tablet_manager.cc:1397] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 12:51:17.846432 22723 log.cc:824] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:17.848079 22724 tablet_bootstrap.cc:492] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: No bootstrap required, opened a new log
I20260924 12:51:17.848354 22724 ts_tablet_manager.cc:1397] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 12:51:17.848919 22723 tablet_bootstrap.cc:492] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: No bootstrap required, opened a new log
I20260924 12:51:17.849197 22723 ts_tablet_manager.cc:1397] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 12:51:17.857587 22723 raft_consensus.cc:348] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.858023 22723 raft_consensus.cc:374] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:17.858116 22723 raft_consensus.cc:729] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9debeba37ccc4264bc8a6ca0d2ff6306, State: Initialized, Role: FOLLOWER
I20260924 12:51:17.857878 22722 raft_consensus.cc:348] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.858330 22722 raft_consensus.cc:374] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:17.858484 22722 raft_consensus.cc:729] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31aee853ad84f8783a221d10be550e4, State: Initialized, Role: FOLLOWER
I20260924 12:51:17.858868 22723 consensus_queue.cc:260] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.859223 22722 consensus_queue.cc:260] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.859198 22724 raft_consensus.cc:348] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.859642 22724 raft_consensus.cc:374] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:17.859761 22724 raft_consensus.cc:729] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b5d3495da0423db03188f0995528b8, State: Initialized, Role: FOLLOWER
I20260924 12:51:17.860203 22574 heartbeater.cc:503] Master 127.2.81.126:37935 was elected leader, sending a full tablet report...
I20260924 12:51:17.860380 22724 consensus_queue.cc:260] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.860852 22723 ts_tablet_manager.cc:1428] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 12:51:17.866460 22704 heartbeater.cc:503] Master 127.2.81.126:37935 was elected leader, sending a full tablet report...
I20260924 12:51:17.866567 22722 ts_tablet_manager.cc:1428] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: Time spent starting tablet: real 0.020s user 0.014s sys 0.003s
I20260924 12:51:17.868063 22724 ts_tablet_manager.cc:1428] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20260924 12:51:17.895741 22730 raft_consensus.cc:482] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 12:51:17.896057 22730 raft_consensus.cc:504] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.897902 22730 leader_election.cc:288] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d31aee853ad84f8783a221d10be550e4 (127.2.81.65:36311), 9debeba37ccc4264bc8a6ca0d2ff6306 (127.2.81.66:35863)
I20260924 12:51:17.904414 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.898419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:32869 (local address 127.2.81.65:36311)
0924 12:51:17.898621 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.898627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.898717 (+ 90us) server_negotiation.cc:407] Connection header received
0924 12:51:17.899049 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.899058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.899142 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.899226 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:17.899787 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.900644 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.902631 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.903473 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.903571 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.903646 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.904051 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.904174 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:51:17.904418 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.898417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.81.67:33239 (local address 127.2.81.66:35863)
0924 12:51:17.898630 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.898636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.899242 (+ 606us) server_negotiation.cc:407] Connection header received
0924 12:51:17.899395 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.899399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.899482 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.899584 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 12:51:17.900198 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.900949 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.902852 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.903626 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.903743 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.903828 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.904054 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.904178 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 12:51:17.904425 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.898296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.2.81.67:32869)
0924 12:51:17.898621 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.898640 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.898804 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.899403 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.899408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.899434 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:17.899613 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.899624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.900770 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.900775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.902457 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.902470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.903664 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.903670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.903849 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.903883 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.903955 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.904087 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":157}
I20260924 12:51:17.904913 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.898399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.66:35863 (local address 127.2.81.67:33239)
0924 12:51:17.899121 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:17.899137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:17.899270 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:17.899668 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:17.899672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:17.899696 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 12:51:17.900032 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.900045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.901115 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.901119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.902671 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:17.902685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.903784 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.903792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:17.903902 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.903923 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 12:51:17.904005 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 12:51:17.904135 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":20,"thread_start_us":423,"threads_started":1}
I20260924 12:51:17.905434 22398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d395cfd3d2b428f94d60e13b7d23ed8" candidate_uuid: "86b5d3495da0423db03188f0995528b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31aee853ad84f8783a221d10be550e4" is_pre_election: true
I20260924 12:51:17.906035 22398 raft_consensus.cc:2454] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b5d3495da0423db03188f0995528b8 in term 0.
I20260924 12:51:17.905928 22529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d395cfd3d2b428f94d60e13b7d23ed8" candidate_uuid: "86b5d3495da0423db03188f0995528b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" is_pre_election: true
I20260924 12:51:17.906433 22529 raft_consensus.cc:2454] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b5d3495da0423db03188f0995528b8 in term 0.
I20260924 12:51:17.906787 22593 leader_election.cc:302] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b5d3495da0423db03188f0995528b8, d31aee853ad84f8783a221d10be550e4; no voters:
I20260924 12:51:17.907196 22730 raft_consensus.cc:2781] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 12:51:17.907321 22730 raft_consensus.cc:482] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 12:51:17.907413 22730 raft_consensus.cc:3037] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:17.909765 22730 raft_consensus.cc:504] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.910405 22730 leader_election.cc:288] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [CANDIDATE]: Term 1 election: Requested vote from peers d31aee853ad84f8783a221d10be550e4 (127.2.81.65:36311), 9debeba37ccc4264bc8a6ca0d2ff6306 (127.2.81.66:35863)
I20260924 12:51:17.910935 22398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d395cfd3d2b428f94d60e13b7d23ed8" candidate_uuid: "86b5d3495da0423db03188f0995528b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31aee853ad84f8783a221d10be550e4"
I20260924 12:51:17.911000 22529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d395cfd3d2b428f94d60e13b7d23ed8" candidate_uuid: "86b5d3495da0423db03188f0995528b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306"
I20260924 12:51:17.911120 22398 raft_consensus.cc:3037] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:17.911221 22529 raft_consensus.cc:3037] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:17.913514 22398 raft_consensus.cc:2454] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b5d3495da0423db03188f0995528b8 in term 1.
I20260924 12:51:17.913534 22529 raft_consensus.cc:2454] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b5d3495da0423db03188f0995528b8 in term 1.
I20260924 12:51:17.913981 22592 leader_election.cc:302] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b5d3495da0423db03188f0995528b8, 9debeba37ccc4264bc8a6ca0d2ff6306; no voters:
I20260924 12:51:17.914336 22730 raft_consensus.cc:2781] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:17.914836 22730 raft_consensus.cc:686] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 1 LEADER]: Becoming Leader. State: Replica: 86b5d3495da0423db03188f0995528b8, State: Running, Role: LEADER
I20260924 12:51:17.915367 22730 consensus_queue.cc:237] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } }
I20260924 12:51:17.920279 22256 catalog_manager.cc:5495] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b5d3495da0423db03188f0995528b8 (127.2.81.67). New cstate: current_term: 1 leader_uuid: "86b5d3495da0423db03188f0995528b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86b5d3495da0423db03188f0995528b8" member_type: VOTER last_known_addr { host: "127.2.81.67" port: 43503 } health_report { overall_health: HEALTHY } } }
I20260924 12:51:17.935990 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.930897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.2.81.67:43503)
0924 12:51:17.931108 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.931112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.931158 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:51:17.931346 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.931350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.931431 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.931564 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:17.931856 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.932620 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.934034 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.934457 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.934556 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:17.935039 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:17.935552 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:17.935684 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:17.935722 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 12:51:17.935788 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 12:51:17.944633 22398 raft_consensus.cc:1261] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 1 FOLLOWER]: Refusing update from remote peer 86b5d3495da0423db03188f0995528b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:17.944744 22529 raft_consensus.cc:1261] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 1 FOLLOWER]: Refusing update from remote peer 86b5d3495da0423db03188f0995528b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:17.945771 22732 consensus_queue.cc:1035] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9debeba37ccc4264bc8a6ca0d2ff6306" member_type: VOTER last_known_addr { host: "127.2.81.66" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:17.946143 22730 consensus_queue.cc:1035] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31aee853ad84f8783a221d10be550e4" member_type: VOTER last_known_addr { host: "127.2.81.65" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:17.977617 22256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:17.977938 22256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 12:51:17.982139 22256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29a6025fa39f4213ba9a3de39dcb3216: Sending DeleteTablet for 3 replicas of tablet 4d395cfd3d2b428f94d60e13b7d23ed8
I20260924 12:51:17.983589 22378 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d395cfd3d2b428f94d60e13b7d23ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:17 UTC) from {username='slave'} at 127.0.0.1:55336
I20260924 12:51:17.983601 22509 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d395cfd3d2b428f94d60e13b7d23ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:17 UTC) from {username='slave'} at 127.0.0.1:54182
I20260924 12:51:17.984019 22639 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d395cfd3d2b428f94d60e13b7d23ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 12:51:17 UTC) from {username='slave'} at 127.0.0.1:40884
I20260924 12:51:17.985052 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22316
I20260924 12:51:17.992592 22745 tablet_replica.cc:330] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: stopping tablet replica
I20260924 12:51:18.001526 22747 tablet_replica.cc:330] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: stopping tablet replica
I20260924 12:51:18.001533 22745 raft_consensus.cc:2229] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:18.002084 22746 tablet_replica.cc:330] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: stopping tablet replica
I20260924 12:51:18.002067 22747 raft_consensus.cc:2229] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 12:51:18.002442 22745 raft_consensus.cc:2258] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:18.002494 22746 raft_consensus.cc:2229] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 12:51:18.002831 22747 raft_consensus.cc:2258] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:18.002980 22746 raft_consensus.cc:2258] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 12:51:18.004320 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:17.986155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.2.81.65:36311)
0924 12:51:17.986346 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:17.986352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:17.992770 (+ 6418us) server_negotiation.cc:407] Connection header received
0924 12:51:17.995013 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:17.995022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:17.995162 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:17.995258 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:17.996035 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:17.997377 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:17.999565 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:18.000343 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:18.000458 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:18.002225 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:18.002237 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:18.002255 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:18.003064 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:18.003132 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:18.003159 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:18.003282 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:18.003840 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 12:51:18.003982 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:51:18.005153 22746 ts_tablet_manager.cc:1905] T 4d395cfd3d2b428f94d60e13b7d23ed8 P d31aee853ad84f8783a221d10be550e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:18.005527 22438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:18.005566 22745 ts_tablet_manager.cc:1905] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:18.005710 22747 ts_tablet_manager.cc:1905] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 12:51:18.013437 22745 ts_tablet_manager.cc:1918] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:18.013747 22745 log.cc:1196] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-1/wal/wals/4d395cfd3d2b428f94d60e13b7d23ed8
I20260924 12:51:18.014317 22747 ts_tablet_manager.cc:1918] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 12:51:18.014532 22747 log.cc:1196] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Deleting WAL directory at /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790253949676453-2373-0/minicluster-data/ts-2/wal/wals/4d395cfd3d2b428f94d60e13b7d23ed8
I20260924 12:51:18.014549 22745 ts_tablet_manager.cc:1939] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 9debeba37ccc4264bc8a6ca0d2ff6306: Deleting consensus metadata
I20260924 12:51:18.015103 22747 ts_tablet_manager.cc:1939] T 4d395cfd3d2b428f94d60e13b7d23ed8 P 86b5d3495da0423db03188f0995528b8: Deleting consensus metadata
I20260924 12:51:18.016889 22241 catalog_manager.cc:4841] TS 9debeba37ccc4264bc8a6ca0d2ff6306 (127.2.81.66:35863): tablet 4d395cfd3d2b428f94d60e13b7d23ed8 (table test-table [id=0ab47b0c06224c2e81279c5eca65c1a5]) successfully deleted
I20260924 12:51:18.017076 22243 catalog_manager.cc:4841] TS 86b5d3495da0423db03188f0995528b8 (127.2.81.67:43503): tablet 4d395cfd3d2b428f94d60e13b7d23ed8 (table test-table [id=0ab47b0c06224c2e81279c5eca65c1a5]) successfully deleted
I20260924 12:51:18.126338 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22316
W20260924 12:51:18.141963 22242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.81.65:36311 (error 108)
I20260924 12:51:18.143306 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22447
I20260924 12:51:18.144127 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.143519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55356)
0924 12:51:18.143711 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.143926 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 12:51:18.144557 22242 catalog_manager.cc:4584] TS d31aee853ad84f8783a221d10be550e4 (127.2.81.65:36311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d395cfd3d2b428f94d60e13b7d23ed8: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
I20260924 12:51:18.151242 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.144339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.2.81.66:35863)
0924 12:51:18.144508 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:18.144513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:18.144529 (+ 16us) server_negotiation.cc:407] Connection header received
0924 12:51:18.144583 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:18.144587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:18.144686 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:18.144780 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:18.145308 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:18.146050 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:18.147686 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:18.148423 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:18.148554 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:18.149864 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:18.149874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:18.149889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:18.150486 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:18.150538 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:18.150564 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:18.150664 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:18.150795 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 12:51:18.150906 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 12:51:18.152040 22569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:18.198549 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.197928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55362)
0924 12:51:18.198225 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.198337 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 12:51:18.199769 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.199349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55376)
0924 12:51:18.199510 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.199605 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 12:51:18.253618 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22447
I20260924 12:51:18.260376 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.259851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55390)
0924 12:51:18.260067 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.260184 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 12:51:18.261880 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.261458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55392)
0924 12:51:18.261628 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.261715 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 12:51:18.270444 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22577
I20260924 12:51:18.278151 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.271282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.2.81.67:43503)
0924 12:51:18.271452 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:18.271458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:18.271477 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:51:18.271666 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:18.271671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:18.271760 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:18.271847 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:18.272257 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:18.273188 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:18.274599 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:18.275168 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:18.275314 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:18.276450 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:18.276458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:18.276475 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:18.277203 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:18.277259 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:18.277337 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:18.277471 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:18.277669 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 12:51:18.277796 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:51:18.279039 22699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:18.342715 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.342219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55408)
0924 12:51:18.342467 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.342560 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 12:51:18.343899 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.343503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55416)
0924 12:51:18.343705 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.343789 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 12:51:18.408727 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22577
I20260924 12:51:18.424436 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22226
I20260924 12:51:18.425634 22287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:18.538136 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:18.537502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.65:36311 (local address 127.0.0.1:55422)
0924 12:51:18.537799 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:18.537900 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.65:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260924 12:51:18.538136 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22226
2026-09-24T12:51:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2838 ms)
[----------] 6 tests from EncryptionPolicyTest (22200 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:18 dist-test-slave-9tm1 krb5kdc[22756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:18 dist-test-slave-9tm1 krb5kdc[22756](info): set up 2 sockets
Sep 24 12:51:18 dist-test-slave-9tm1 krb5kdc[22756](info): commencing operation
krb5kdc: starting...
W20260924 12:51:19.581532 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:19 dist-test-slave-9tm1 krb5kdc[22756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:19.836735 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:20.171566 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:20.171868 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:20.171960 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:20.179466 22769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:20.179595 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:20.179641 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:20.179704 22769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:20.179747 22769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:20.190004 22769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:20.191303 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:20.192588 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:20.204254 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:20.204237 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:20.204183 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:20.205226 22769 server_base.cc:1034] running on GCE node
I20260924 12:51:20.206012 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:20.207428 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:20.208624 22769 hybrid_clock.cc:648] HybridClock initialized: now 1790254280208578 us; error 37 us; skew 500 ppm
I20260924 12:51:20.208956 22769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:20 dist-test-slave-9tm1 krb5kdc[22756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:20.216809 22769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:20.218550 22769 webserver.cc:460] Webserver started at http://127.0.0.1:42945/ using document root <none> and password file <none>
I20260924 12:51:20.219298 22769 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:20.219421 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:20.219751 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:20.222906 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "0d35ee67e6254fe7a778ed66f2bd0669"
format_stamp: "Formatted at 2026-09-24 12:51:20 on dist-test-slave-9tm1"
server_key: "3e9aa154462e1c7ca03e4a3b1e11c2ce"
server_key_iv: "3f146f89ff1c8e1dd368976ba0605246"
server_key_version: "encryptionkey@0"
I20260924 12:51:20.223680 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "0d35ee67e6254fe7a778ed66f2bd0669"
format_stamp: "Formatted at 2026-09-24 12:51:20 on dist-test-slave-9tm1"
server_key: "3e9aa154462e1c7ca03e4a3b1e11c2ce"
server_key_iv: "3f146f89ff1c8e1dd368976ba0605246"
server_key_version: "encryptionkey@0"
I20260924 12:51:20.229411 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 12:51:20.233459 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:20.234954 22769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:51:20.235153 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "0d35ee67e6254fe7a778ed66f2bd0669"
format_stamp: "Formatted at 2026-09-24 12:51:20 on dist-test-slave-9tm1"
server_key: "3e9aa154462e1c7ca03e4a3b1e11c2ce"
server_key_iv: "3f146f89ff1c8e1dd368976ba0605246"
server_key_version: "encryptionkey@0"
I20260924 12:51:20.235349 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:20.254886 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:20.259070 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:20.259325 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:20.277065 22769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39727
I20260924 12:51:20.277084 22835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39727 every 8 connection(s)
I20260924 12:51:20.278523 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:20.281124 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22769
I20260924 12:51:20.281339 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:20.282104 2373 external_mini_cluster.cc:1428] Setting key 14b08b7e6c0436568a146011343be8e4
I20260924 12:51:20.283126 22836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:51:20 dist-test-slave-9tm1 krb5kdc[22756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 12:51:20.298431 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669: Bootstrap starting.
I20260924 12:51:20.302263 22836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:20.303447 22836 log.cc:824] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:20.306890 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669: No bootstrap required, opened a new log
I20260924 12:51:20.307667 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:20.284180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33603 (local address 127.0.0.1:39727)
0924 12:51:20.285263 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:20.285276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:20.285319 (+ 43us) server_negotiation.cc:407] Connection header received
0924 12:51:20.287052 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:20.287081 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:20.287577 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:20.287969 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:20.289901 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:20.291371 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:20.293265 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:20.294059 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:20.294233 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:20.299509 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:20.299524 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:20.299541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:20.299642 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:20.302991 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:20.304107 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:20.304120 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:20.304129 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:20.304377 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:20.304917 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:20.304923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:20.304926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:20.305523 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:20.305771 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:20.306258 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 12:51:20.306546 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":397,"threads_started":1}
I20260924 12:51:20.311625 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:20.316324 22836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260924 12:51:20.316871 22836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:20.317021 22836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d35ee67e6254fe7a778ed66f2bd0669, State: Initialized, Role: FOLLOWER
I20260924 12:51:20.317786 22836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260924 12:51:20.318269 22836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:20.318437 22836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:20.318756 22836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:20.321833 22836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260924 12:51:20.322510 22836 leader_election.cc:302] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d35ee67e6254fe7a778ed66f2bd0669; no voters:
I20260924 12:51:20.323239 22836 leader_election.cc:288] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:20.323338 22846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:20.324573 22846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [term 1 LEADER]: Becoming Leader. State: Replica: 0d35ee67e6254fe7a778ed66f2bd0669, State: Running, Role: LEADER
I20260924 12:51:20.325309 22846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } }
I20260924 12:51:20.325716 22836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:20.334717 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } } }
I20260924 12:51:20.335182 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:20.334694 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d35ee67e6254fe7a778ed66f2bd0669. Latest consensus state: current_term: 1 leader_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d35ee67e6254fe7a778ed66f2bd0669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39727 } } }
I20260924 12:51:20.335309 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:20.336014 22860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:20.336790 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:20.317451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56613 (local address 127.0.0.1:39727)
0924 12:51:20.317707 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:20.317713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:20.318291 (+ 578us) server_negotiation.cc:407] Connection header received
0924 12:51:20.318476 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:20.318482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:20.318581 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:20.318715 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:20.320784 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:20.321638 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:20.323481 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:20.324345 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:20.324496 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:20.327999 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:20.328006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:20.328012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:20.328073 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:20.332173 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:20.333388 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:20.333394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:20.333397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:20.333608 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:20.334229 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:20.334234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:20.334237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:20.334651 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:20.335978 (+ 1327us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:20.336190 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 12:51:20.336383 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 12:51:20.341391 22860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:20.349730 22860 catalog_manager.cc:1347] Generated new cluster ID: c3419b8169024726b881db50f4696f87
I20260924 12:51:20.349860 22860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:20.374673 22860 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:20.376619 22860 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:20.389681 22860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d35ee67e6254fe7a778ed66f2bd0669: Generated new TSK 0
I20260924 12:51:20.390790 22860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:20.403751 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22769
I20260924 12:51:20.405025 22831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:20.499559 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22769
W20260924 12:51:20.516222 22754 connection.cc:535] client connection to 127.0.0.1:39727 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39727): Connection reset by peer (error 104)
2026-09-24T12:51:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:20 dist-test-slave-9tm1 krb5kdc[22868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:20 dist-test-slave-9tm1 krb5kdc[22868](info): set up 2 sockets
Sep 24 12:51:20 dist-test-slave-9tm1 krb5kdc[22868](info): commencing operation
krb5kdc: starting...
W20260924 12:51:21.533313 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:21 dist-test-slave-9tm1 krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:21.808707 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:22.147367 22881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:22.147693 22881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:22.147818 22881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:22.155411 22881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:22.155553 22881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:22.155629 22881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:22.155710 22881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:22.155773 22881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:22.166018 22881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:22.167331 22881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:22.168643 22881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:22.180388 22886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:22.180356 22887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:22.180437 22889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:22.181090 22881 server_base.cc:1034] running on GCE node
I20260924 12:51:22.181903 22881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:22.183375 22881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:22.184651 22881 hybrid_clock.cc:648] HybridClock initialized: now 1790254282184598 us; error 38 us; skew 500 ppm
I20260924 12:51:22.185086 22881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:22 dist-test-slave-9tm1 krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:22.188555 22881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:22.190325 22881 webserver.cc:460] Webserver started at http://127.0.0.1:36725/ using document root <none> and password file <none>
I20260924 12:51:22.191103 22881 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:22.191241 22881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:22.191565 22881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:22.195084 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "ef512de587de4ad3b91f49ddb86eb088"
format_stamp: "Formatted at 2026-09-24 12:51:22 on dist-test-slave-9tm1"
server_key: "ecbfc46f6703ca6ffd8202cba517baff"
server_key_iv: "86dc101f1d14dc04f9da9e67b96702bf"
server_key_version: "encryptionkey@0"
I20260924 12:51:22.195927 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "ef512de587de4ad3b91f49ddb86eb088"
format_stamp: "Formatted at 2026-09-24 12:51:22 on dist-test-slave-9tm1"
server_key: "ecbfc46f6703ca6ffd8202cba517baff"
server_key_iv: "86dc101f1d14dc04f9da9e67b96702bf"
server_key_version: "encryptionkey@0"
I20260924 12:51:22.201807 22881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 12:51:22.205891 22896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:22.207505 22881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260924 12:51:22.207713 22881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "ef512de587de4ad3b91f49ddb86eb088"
format_stamp: "Formatted at 2026-09-24 12:51:22 on dist-test-slave-9tm1"
server_key: "ecbfc46f6703ca6ffd8202cba517baff"
server_key_iv: "86dc101f1d14dc04f9da9e67b96702bf"
server_key_version: "encryptionkey@0"
I20260924 12:51:22.207921 22881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:22.234423 22881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:22.238680 22881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:22.238955 22881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:22.256659 22881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38013
I20260924 12:51:22.256685 22947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38013 every 8 connection(s)
I20260924 12:51:22.258072 22881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:22.261999 22948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:22.263029 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22881
I20260924 12:51:22.263221 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:22.263743 2373 external_mini_cluster.cc:1428] Setting key c695ee454d29e045d7a828e18f3d90d5
I20260924 12:51:22.275967 22948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088: Bootstrap starting.
I20260924 12:51:22.277544 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:22.265608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43303 (local address 127.0.0.1:38013)
0924 12:51:22.266371 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:22.266383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:22.266428 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:51:22.268262 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:22.268286 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:22.268722 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:22.269097 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:22.269678 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:22.270938 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:22.272836 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:22.273679 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:22.273819 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:22.275128 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:22.275136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:22.275151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:22.275699 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:22.275783 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:22.275810 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:22.275928 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:22.276275 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 12:51:22.276471 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":354,"threads_started":1}
I20260924 12:51:22.279392 22948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:22.280550 22948 log.cc:824] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:22.281571 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:22.283305 22948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088: No bootstrap required, opened a new log
I20260924 12:51:22.291497 22948 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef512de587de4ad3b91f49ddb86eb088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 12:51:22.291885 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:22.285988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41801 (local address 127.0.0.1:38013)
0924 12:51:22.286219 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:22.286225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:22.286412 (+ 187us) server_negotiation.cc:407] Connection header received
0924 12:51:22.286645 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:22.286649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:22.286713 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:22.286808 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:22.287310 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:22.288087 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:22.289463 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:22.289957 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:22.290083 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:22.291183 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:22.291188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:22.291191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:22.291209 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:22.291266 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:22.291284 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:22.291374 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:22.291565 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 12:51:22.291632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 12:51:22.292027 22948 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:22.292142 22948 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef512de587de4ad3b91f49ddb86eb088, State: Initialized, Role: FOLLOWER
I20260924 12:51:22.292831 22948 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef512de587de4ad3b91f49ddb86eb088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 12:51:22.293160 22948 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:22.293318 22948 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:22.293529 22948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:22.295853 22948 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef512de587de4ad3b91f49ddb86eb088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 12:51:22.296420 22948 leader_election.cc:302] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef512de587de4ad3b91f49ddb86eb088; no voters:
I20260924 12:51:22.297051 22948 leader_election.cc:288] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:22.297163 22958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:22.297885 22958 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [term 1 LEADER]: Becoming Leader. State: Replica: ef512de587de4ad3b91f49ddb86eb088, State: Running, Role: LEADER
I20260924 12:51:22.298460 22958 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef512de587de4ad3b91f49ddb86eb088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } }
I20260924 12:51:22.299093 22948 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:22.301303 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef512de587de4ad3b91f49ddb86eb088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef512de587de4ad3b91f49ddb86eb088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } } }
I20260924 12:51:22.301282 22960 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef512de587de4ad3b91f49ddb86eb088. Latest consensus state: current_term: 1 leader_uuid: "ef512de587de4ad3b91f49ddb86eb088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef512de587de4ad3b91f49ddb86eb088" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38013 } } }
I20260924 12:51:22.301903 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:22.301923 22960 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:22.306598 22966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:22.312282 22966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:22.321602 22966 catalog_manager.cc:1347] Generated new cluster ID: 910ee54afb8c4faaa2ad950b67a461fe
I20260924 12:51:22.321733 22966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:22.347335 22966 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:22.349287 22966 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:22.362920 22966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef512de587de4ad3b91f49ddb86eb088: Generated new TSK 0
I20260924 12:51:22.364068 22966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:22.374411 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22881
W20260924 12:51:22.375813 22943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43303
W20260924 12:51:22.376614 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22881: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:22.376775 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22881
W20260924 12:51:22.390601 22866 connection.cc:535] client connection to 127.0.0.1:38013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38013): Connection reset by peer (error 104)
2026-09-24T12:51:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:22 dist-test-slave-9tm1 krb5kdc[22979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:22 dist-test-slave-9tm1 krb5kdc[22979](info): set up 2 sockets
Sep 24 12:51:22 dist-test-slave-9tm1 krb5kdc[22979](info): commencing operation
krb5kdc: starting...
W20260924 12:51:23.421326 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:23 dist-test-slave-9tm1 krb5kdc[22979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:23.689132 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:24.032478 22992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:24.032806 22992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:24.032930 22992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:24.040606 22992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:24.040742 22992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:24.040791 22992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:24.040876 22992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:24.040934 22992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:24.051367 22992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:24.052888 22992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:24.054396 22992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:24.067605 22997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:24.067662 22998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:24.067631 23000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:24.068174 22992 server_base.cc:1034] running on GCE node
I20260924 12:51:24.069135 22992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:24.070664 22992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:24.071923 22992 hybrid_clock.cc:648] HybridClock initialized: now 1790254284071858 us; error 58 us; skew 500 ppm
I20260924 12:51:24.072335 22992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:24 dist-test-slave-9tm1 krb5kdc[22979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:24.076303 22992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:24.078161 22992 webserver.cc:460] Webserver started at http://127.0.0.1:44367/ using document root <none> and password file <none>
I20260924 12:51:24.079011 22992 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:24.079149 22992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:24.079448 22992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:24.082685 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "0cc6199d74c6411d87f9a33cff1d9410"
format_stamp: "Formatted at 2026-09-24 12:51:24 on dist-test-slave-9tm1"
server_key: "78ccb8c4e329ee23cc0c7b7396576e5a"
server_key_iv: "7746d0bf6fa4682d46e5934d51ab5611"
server_key_version: "encryptionkey@0"
I20260924 12:51:24.083535 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "0cc6199d74c6411d87f9a33cff1d9410"
format_stamp: "Formatted at 2026-09-24 12:51:24 on dist-test-slave-9tm1"
server_key: "78ccb8c4e329ee23cc0c7b7396576e5a"
server_key_iv: "7746d0bf6fa4682d46e5934d51ab5611"
server_key_version: "encryptionkey@0"
I20260924 12:51:24.089460 22992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 12:51:24.093897 23007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:24.096182 22992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 12:51:24.096418 22992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "0cc6199d74c6411d87f9a33cff1d9410"
format_stamp: "Formatted at 2026-09-24 12:51:24 on dist-test-slave-9tm1"
server_key: "78ccb8c4e329ee23cc0c7b7396576e5a"
server_key_iv: "7746d0bf6fa4682d46e5934d51ab5611"
server_key_version: "encryptionkey@0"
I20260924 12:51:24.096642 22992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:24.103578 22992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:24.109424 22992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:24.109923 22992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:24.131361 22992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34097
I20260924 12:51:24.131374 23058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34097 every 8 connection(s)
I20260924 12:51:24.132889 22992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:24.136014 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 22992
I20260924 12:51:24.136183 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:24.136646 2373 external_mini_cluster.cc:1428] Setting key 52e692eec903c409e6265159bc7d4470
I20260924 12:51:24.137185 23059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:24.146276 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:24.138487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54077 (local address 127.0.0.1:34097)
0924 12:51:24.139473 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:24.139490 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:24.139763 (+ 273us) server_negotiation.cc:407] Connection header received
0924 12:51:24.142285 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:24.142314 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:24.142750 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:24.143159 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:24.144361 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:24.144368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:24.144381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:24.144397 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:24.144466 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:24.144488 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:24.144713 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:24.144830 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 12:51:24.145084 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":395,"threads_started":1}
I20260924 12:51:24.151412 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:24.154644 23059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410: Bootstrap starting.
I20260924 12:51:24.160048 23059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:24.161598 23059 log.cc:824] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:24.162128 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:24.158830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34299 (local address 127.0.0.1:34097)
0924 12:51:24.159056 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:24.159062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:24.159700 (+ 638us) server_negotiation.cc:407] Connection header received
0924 12:51:24.159937 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:24.159941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:24.160004 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:24.160106 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:24.161309 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:24.161314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:24.161318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:24.161336 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:24.161384 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:24.161401 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:24.161511 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:24.161720 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 12:51:24.161886 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 12:51:24.165072 23059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410: No bootstrap required, opened a new log
I20260924 12:51:24.175114 23059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc6199d74c6411d87f9a33cff1d9410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260924 12:51:24.175527 23059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:24.175623 23059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc6199d74c6411d87f9a33cff1d9410, State: Initialized, Role: FOLLOWER
I20260924 12:51:24.176236 23059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc6199d74c6411d87f9a33cff1d9410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260924 12:51:24.176448 23059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:24.176581 23059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:24.176731 23059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:24.179241 23059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc6199d74c6411d87f9a33cff1d9410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260924 12:51:24.179872 23059 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc6199d74c6411d87f9a33cff1d9410; no voters:
I20260924 12:51:24.180546 23059 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:24.180683 23069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:24.181332 23069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [term 1 LEADER]: Becoming Leader. State: Replica: 0cc6199d74c6411d87f9a33cff1d9410, State: Running, Role: LEADER
I20260924 12:51:24.182015 23069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc6199d74c6411d87f9a33cff1d9410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260924 12:51:24.182592 23059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:24.185251 23070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc6199d74c6411d87f9a33cff1d9410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc6199d74c6411d87f9a33cff1d9410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } } }
I20260924 12:51:24.185611 23070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:24.186280 23071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc6199d74c6411d87f9a33cff1d9410. Latest consensus state: current_term: 1 leader_uuid: "0cc6199d74c6411d87f9a33cff1d9410" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc6199d74c6411d87f9a33cff1d9410" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } } }
I20260924 12:51:24.186579 23071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:24.187414 23076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:24.194880 23076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:24.205801 23076 catalog_manager.cc:1347] Generated new cluster ID: 192757e3bbac4d6a86c7616010b95607
I20260924 12:51:24.205931 23076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:24.218000 23076 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:24.218508 23076 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:24.235713 23076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc6199d74c6411d87f9a33cff1d9410: Generated new TSK 0
I20260924 12:51:24.236665 23076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:24.244565 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22992
W20260924 12:51:24.245695 23054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54077
W20260924 12:51:24.246496 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 22992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:24.246625 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 22992
2026-09-24T12:51:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:24 dist-test-slave-9tm1 krb5kdc[23090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:24 dist-test-slave-9tm1 krb5kdc[23090](info): set up 2 sockets
Sep 24 12:51:24 dist-test-slave-9tm1 krb5kdc[23090](info): commencing operation
krb5kdc: starting...
W20260924 12:51:25.297389 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:25 dist-test-slave-9tm1 krb5kdc[23090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:25.578220 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:25.928947 23103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:25.929323 23103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:25.929453 23103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:25.936880 23103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:25.937028 23103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:25.937100 23103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:25.937182 23103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:25.937229 23103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:25.947449 23103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:25.948791 23103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:25.950103 23103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:25.962147 23108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:25.962268 23109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:25.962193 23111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:25.963490 23103 server_base.cc:1034] running on GCE node
I20260924 12:51:25.964370 23103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:25.966046 23103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:25.967303 23103 hybrid_clock.cc:648] HybridClock initialized: now 1790254285967262 us; error 35 us; skew 500 ppm
I20260924 12:51:25.967698 23103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:25 dist-test-slave-9tm1 krb5kdc[23090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:25.978706 23103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:25.980438 23103 webserver.cc:460] Webserver started at http://127.0.0.1:43553/ using document root <none> and password file <none>
I20260924 12:51:25.981232 23103 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:25.981353 23103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:25.981683 23103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:25.984891 23103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "79e5129972254997882ef9f4d0057bc2"
format_stamp: "Formatted at 2026-09-24 12:51:25 on dist-test-slave-9tm1"
server_key: "c6366c7cdaa4cf5c2c0bc7d61f954419"
server_key_iv: "5341023fe3db9495672a9e38e94ec29c"
server_key_version: "encryptionkey@0"
I20260924 12:51:25.985776 23103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "79e5129972254997882ef9f4d0057bc2"
format_stamp: "Formatted at 2026-09-24 12:51:25 on dist-test-slave-9tm1"
server_key: "c6366c7cdaa4cf5c2c0bc7d61f954419"
server_key_iv: "5341023fe3db9495672a9e38e94ec29c"
server_key_version: "encryptionkey@0"
I20260924 12:51:25.991209 23103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:25.995296 23118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:25.996944 23103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:25.997190 23103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "79e5129972254997882ef9f4d0057bc2"
format_stamp: "Formatted at 2026-09-24 12:51:25 on dist-test-slave-9tm1"
server_key: "c6366c7cdaa4cf5c2c0bc7d61f954419"
server_key_iv: "5341023fe3db9495672a9e38e94ec29c"
server_key_version: "encryptionkey@0"
I20260924 12:51:25.997412 23103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:26.025854 23103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:26.030443 23103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:26.030732 23103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:26.050364 23103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33455
I20260924 12:51:26.050381 23169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33455 every 8 connection(s)
I20260924 12:51:26.051929 23103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:26.056172 23170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:26.057157 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23103
I20260924 12:51:26.057330 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:26.057782 2373 external_mini_cluster.cc:1428] Setting key ec1c4656f08ee5760621edfc35bf6e33
I20260924 12:51:26.070252 23170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2: Bootstrap starting.
Sep 24 12:51:26 dist-test-slave-9tm1 krb5kdc[23090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 12:51:26.075110 23170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:26.076399 23170 log.cc:824] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:26.080036 23170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2: No bootstrap required, opened a new log
I20260924 12:51:26.084961 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:26.059871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50217 (local address 127.0.0.1:33455)
0924 12:51:26.060737 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:26.060757 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:26.060808 (+ 51us) server_negotiation.cc:407] Connection header received
0924 12:51:26.063275 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:26.063307 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:26.063803 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:26.064209 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:26.065789 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:26.067135 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:26.068719 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:26.069513 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:26.069717 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:26.075335 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:26.075346 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:26.075367 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:26.075414 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:26.079046 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:26.080184 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:26.080194 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:26.080204 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:26.080408 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:26.081005 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:26.081011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:26.081015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:26.081979 (+ 964us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:26.082491 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:26.083173 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 12:51:26.083782 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":364,"threads_started":1}
I20260924 12:51:26.089711 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:26.090119 23170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e5129972254997882ef9f4d0057bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260924 12:51:26.090515 23170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:26.090621 23170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e5129972254997882ef9f4d0057bc2, State: Initialized, Role: FOLLOWER
I20260924 12:51:26.091202 23170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e5129972254997882ef9f4d0057bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260924 12:51:26.091425 23170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:26.091562 23170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:26.091725 23170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:26.094341 23170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e5129972254997882ef9f4d0057bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260924 12:51:26.095052 23170 leader_election.cc:302] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e5129972254997882ef9f4d0057bc2; no voters:
I20260924 12:51:26.095703 23170 leader_election.cc:288] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:26.095839 23180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:26.098320 23180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 79e5129972254997882ef9f4d0057bc2, State: Running, Role: LEADER
I20260924 12:51:26.098320 23170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:26.099007 23180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e5129972254997882ef9f4d0057bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260924 12:51:26.103101 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e5129972254997882ef9f4d0057bc2. Latest consensus state: current_term: 1 leader_uuid: "79e5129972254997882ef9f4d0057bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e5129972254997882ef9f4d0057bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } } }
I20260924 12:51:26.103579 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:26.104194 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e5129972254997882ef9f4d0057bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e5129972254997882ef9f4d0057bc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } } }
I20260924 12:51:26.104478 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:26.115343 23190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:26.115999 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:26.094498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49075 (local address 127.0.0.1:33455)
0924 12:51:26.094684 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:26.094690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:26.095748 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 12:51:26.096108 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:26.096114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:26.096218 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:26.096329 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:26.098115 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:26.099491 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:26.101701 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:26.102431 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:26.102573 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:26.106036 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:26.106044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:26.106050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:26.106113 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:26.111653 (+ 5540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:26.113131 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:26.113138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:26.113142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:26.113336 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:26.113867 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:26.113872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:26.113880 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:26.114377 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:26.114644 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:26.115280 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 12:51:26.115581 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 12:51:26.120841 23190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:26.130475 23190 catalog_manager.cc:1347] Generated new cluster ID: d3a01fa797664c008639298dfb786c4c
I20260924 12:51:26.130617 23190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:26.159642 23190 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:26.161756 23190 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:26.171578 23190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e5129972254997882ef9f4d0057bc2: Generated new TSK 0
I20260924 12:51:26.172776 23190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:26.179854 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23103
I20260924 12:51:26.181176 23165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:26.279089 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23103
2026-09-24T12:51:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:26 dist-test-slave-9tm1 krb5kdc[23202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:26 dist-test-slave-9tm1 krb5kdc[23202](info): set up 2 sockets
Sep 24 12:51:26 dist-test-slave-9tm1 krb5kdc[23202](info): commencing operation
krb5kdc: starting...
W20260924 12:51:27.341606 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:27 dist-test-slave-9tm1 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:27.621857 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:28.001783 23215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:28.002146 23215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:28.002290 23215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:28.010075 23215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:28.010236 23215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:28.010370 23215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:28.010448 23215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:28.010488 23215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:28.022917 23215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:28.024355 23215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:28.025811 23215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:28.039573 23221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:28.039783 23220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:28.040192 23223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:28.040894 23215 server_base.cc:1034] running on GCE node
I20260924 12:51:28.041843 23215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:28.043474 23215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:28.045051 23215 hybrid_clock.cc:648] HybridClock initialized: now 1790254288044798 us; error 55 us; skew 500 ppm
I20260924 12:51:28.045523 23215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:28 dist-test-slave-9tm1 krb5kdc[23202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:28.049006 23215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:28.050752 23215 webserver.cc:460] Webserver started at http://127.0.0.1:45885/ using document root <none> and password file <none>
I20260924 12:51:28.051520 23215 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:28.051661 23215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:28.051964 23215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:28.055279 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "3c70c97487a64247bee4097c66502796"
format_stamp: "Formatted at 2026-09-24 12:51:28 on dist-test-slave-9tm1"
server_key: "2c3e5c5920e0a66ea91801a552857da3"
server_key_iv: "27e0b46d164a7a2d1e16c15eef5ce377"
server_key_version: "encryptionkey@0"
I20260924 12:51:28.056155 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "3c70c97487a64247bee4097c66502796"
format_stamp: "Formatted at 2026-09-24 12:51:28 on dist-test-slave-9tm1"
server_key: "2c3e5c5920e0a66ea91801a552857da3"
server_key_iv: "27e0b46d164a7a2d1e16c15eef5ce377"
server_key_version: "encryptionkey@0"
I20260924 12:51:28.063669 23215 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 12:51:28.068661 23230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:28.070541 23215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 12:51:28.070744 23215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "3c70c97487a64247bee4097c66502796"
format_stamp: "Formatted at 2026-09-24 12:51:28 on dist-test-slave-9tm1"
server_key: "2c3e5c5920e0a66ea91801a552857da3"
server_key_iv: "27e0b46d164a7a2d1e16c15eef5ce377"
server_key_version: "encryptionkey@0"
I20260924 12:51:28.071087 23215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:28.090035 23215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:28.095311 23215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:28.095628 23215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:28.115346 23215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40035
I20260924 12:51:28.115372 23281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40035 every 8 connection(s)
I20260924 12:51:28.117229 23215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:28.119410 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23215
I20260924 12:51:28.119608 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:28.120163 2373 external_mini_cluster.cc:1428] Setting key 061476730aca8c4483322b8f78af5789
I20260924 12:51:28.122490 23282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:28.136957 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:28.122191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52509 (local address 127.0.0.1:40035)
0924 12:51:28.123043 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:28.123065 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:28.123112 (+ 47us) server_negotiation.cc:407] Connection header received
0924 12:51:28.125200 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:28.125228 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:28.125740 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:28.126167 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:28.127058 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:28.128702 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:28.130871 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:28.131645 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:28.131817 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:28.133719 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:28.133735 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:28.133760 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:28.134440 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:28.134618 (+ 178us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:28.134652 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:28.134768 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:28.135339 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 12:51:28.135773 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":397,"threads_started":1}
I20260924 12:51:28.137907 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796: Bootstrap starting.
I20260924 12:51:28.141934 23282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:28.142005 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:28.143076 23282 log.cc:824] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:28.146586 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796: No bootstrap required, opened a new log
I20260924 12:51:28.153789 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:28.146721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57009 (local address 127.0.0.1:40035)
0924 12:51:28.146934 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:28.146940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:28.147328 (+ 388us) server_negotiation.cc:407] Connection header received
0924 12:51:28.147541 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:28.147545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:28.147620 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:28.147710 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:28.148456 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:28.149439 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:28.150852 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:28.151372 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:28.151505 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:28.152901 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:28.152908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:28.152912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:28.152936 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:28.153028 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:28.153053 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:28.153153 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:28.153298 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 12:51:28.153409 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:51:28.157044 23282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c70c97487a64247bee4097c66502796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40035 } }
I20260924 12:51:28.157469 23282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:28.157547 23282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c70c97487a64247bee4097c66502796, State: Initialized, Role: FOLLOWER
I20260924 12:51:28.158205 23282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c70c97487a64247bee4097c66502796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40035 } }
I20260924 12:51:28.158504 23282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:28.158699 23282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:28.159080 23282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:28.161805 23282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c70c97487a64247bee4097c66502796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40035 } }
I20260924 12:51:28.162612 23282 leader_election.cc:302] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c70c97487a64247bee4097c66502796; no voters:
I20260924 12:51:28.163286 23282 leader_election.cc:288] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:28.163442 23292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:28.164587 23292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [term 1 LEADER]: Becoming Leader. State: Replica: 3c70c97487a64247bee4097c66502796, State: Running, Role: LEADER
I20260924 12:51:28.165321 23292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c70c97487a64247bee4097c66502796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40035 } }
I20260924 12:51:28.165736 23282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:28.168488 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c70c97487a64247bee4097c66502796. Latest consensus state: current_term: 1 leader_uuid: "3c70c97487a64247bee4097c66502796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c70c97487a64247bee4097c66502796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40035 } } }
I20260924 12:51:28.169168 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:28.168634 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c70c97487a64247bee4097c66502796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c70c97487a64247bee4097c66502796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40035 } } }
I20260924 12:51:28.169699 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:28.173635 23301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:28.181981 23301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:28.194015 23301 catalog_manager.cc:1347] Generated new cluster ID: 60664cf2599c4473a5d339ee6fcb357f
I20260924 12:51:28.194245 23301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:28.215139 23301 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:28.217437 23301 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:28.230135 23301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c70c97487a64247bee4097c66502796: Generated new TSK 0
I20260924 12:51:28.231138 23301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:28.245705 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23215
W20260924 12:51:28.246980 23277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52509
W20260924 12:51:28.247794 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:28.247969 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23215
2026-09-24T12:51:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:28 dist-test-slave-9tm1 krb5kdc[23313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:28 dist-test-slave-9tm1 krb5kdc[23313](info): set up 2 sockets
Sep 24 12:51:28 dist-test-slave-9tm1 krb5kdc[23313](info): commencing operation
krb5kdc: starting...
W20260924 12:51:29.313586 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:29 dist-test-slave-9tm1 krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:29.599195 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:29.949299 23326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:29.949615 23326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:29.949750 23326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:29.957482 23326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:29.957619 23326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:29.957695 23326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:29.957782 23326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:29.957839 23326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:29.968858 23326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:29.970244 23326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:29.971591 23326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:29.984732 23331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:29.984724 23332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:29.984946 23334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:29.985699 23326 server_base.cc:1034] running on GCE node
I20260924 12:51:29.986647 23326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:29.988361 23326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:29.990024 23326 hybrid_clock.cc:648] HybridClock initialized: now 1790254289989916 us; error 95 us; skew 500 ppm
I20260924 12:51:29.990451 23326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:29 dist-test-slave-9tm1 krb5kdc[23313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:29.994792 23326 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:29.996654 23326 webserver.cc:460] Webserver started at http://127.0.0.1:37569/ using document root <none> and password file <none>
I20260924 12:51:29.997478 23326 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:29.997629 23326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:29.997956 23326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:30.001487 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "2975618b685a4159a08415f5900dd146"
format_stamp: "Formatted at 2026-09-24 12:51:29 on dist-test-slave-9tm1"
server_key: "802cc4631761518f000d8c2663e23984"
server_key_iv: "6cd2488201d259b0614e3671ff6bba14"
server_key_version: "encryptionkey@0"
I20260924 12:51:30.002465 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "2975618b685a4159a08415f5900dd146"
format_stamp: "Formatted at 2026-09-24 12:51:29 on dist-test-slave-9tm1"
server_key: "802cc4631761518f000d8c2663e23984"
server_key_iv: "6cd2488201d259b0614e3671ff6bba14"
server_key_version: "encryptionkey@0"
I20260924 12:51:30.009537 23326 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 12:51:30.014309 23341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:30.017043 23326 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:30.017344 23326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "2975618b685a4159a08415f5900dd146"
format_stamp: "Formatted at 2026-09-24 12:51:29 on dist-test-slave-9tm1"
server_key: "802cc4631761518f000d8c2663e23984"
server_key_iv: "6cd2488201d259b0614e3671ff6bba14"
server_key_version: "encryptionkey@0"
I20260924 12:51:30.017587 23326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:30.025744 23326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:30.030849 23326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:30.031175 23326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:30.050709 23326 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42729
I20260924 12:51:30.050750 23392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42729 every 8 connection(s)
I20260924 12:51:30.052143 23326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:30.055995 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23326
I20260924 12:51:30.056166 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:30.056159 23393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:30.056605 2373 external_mini_cluster.cc:1428] Setting key aa06ee493d4b7ba52a27a60c49c813ae
I20260924 12:51:30.066776 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:30.058530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59821 (local address 127.0.0.1:42729)
0924 12:51:30.059838 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:30.059853 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:30.059907 (+ 54us) server_negotiation.cc:407] Connection header received
0924 12:51:30.061823 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:30.061854 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:30.062340 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:30.062742 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:30.064670 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:30.064681 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:30.064705 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:30.064728 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:30.064805 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:30.064838 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:30.065109 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:30.065239 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 12:51:30.065625 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":658,"threads_started":1}
I20260924 12:51:30.072067 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:30.074281 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146: Bootstrap starting.
I20260924 12:51:30.079761 23393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:30.080964 23393 log.cc:824] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:30.081915 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:30.078959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53153 (local address 127.0.0.1:42729)
0924 12:51:30.079195 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:30.079201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:30.079509 (+ 308us) server_negotiation.cc:407] Connection header received
0924 12:51:30.079952 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:30.079958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:30.080026 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:30.080119 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:30.081088 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:30.081093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:30.081099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:30.081117 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:30.081162 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:30.081184 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:30.081305 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:30.081401 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 12:51:30.081475 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:51:30.084695 23393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146: No bootstrap required, opened a new log
I20260924 12:51:30.093048 23393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2975618b685a4159a08415f5900dd146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260924 12:51:30.093626 23393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:30.093753 23393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2975618b685a4159a08415f5900dd146, State: Initialized, Role: FOLLOWER
I20260924 12:51:30.094456 23393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2975618b685a4159a08415f5900dd146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260924 12:51:30.094708 23393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:30.094839 23393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:30.094992 23393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:30.097512 23393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2975618b685a4159a08415f5900dd146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260924 12:51:30.098120 23393 leader_election.cc:302] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2975618b685a4159a08415f5900dd146; no voters:
I20260924 12:51:30.098953 23393 leader_election.cc:288] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:30.099089 23403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:30.100224 23403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [term 1 LEADER]: Becoming Leader. State: Replica: 2975618b685a4159a08415f5900dd146, State: Running, Role: LEADER
I20260924 12:51:30.101264 23403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2975618b685a4159a08415f5900dd146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } }
I20260924 12:51:30.103008 23393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:30.104772 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2975618b685a4159a08415f5900dd146. Latest consensus state: current_term: 1 leader_uuid: "2975618b685a4159a08415f5900dd146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2975618b685a4159a08415f5900dd146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } } }
I20260924 12:51:30.105302 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:30.104872 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2975618b685a4159a08415f5900dd146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2975618b685a4159a08415f5900dd146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42729 } } }
I20260924 12:51:30.106154 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:30.110224 23408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:30.117550 23408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:30.129896 23408 catalog_manager.cc:1347] Generated new cluster ID: d58dcb6b41a54ca698737a407688cb7d
I20260924 12:51:30.130134 23408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:30.146590 23408 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:30.147199 23408 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:30.167771 23408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2975618b685a4159a08415f5900dd146: Generated new TSK 0
I20260924 12:51:30.169132 23408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:30.184660 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23326
W20260924 12:51:30.186014 23388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59821
W20260924 12:51:30.186820 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23326: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:30.187017 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23326
2026-09-24T12:51:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:30 dist-test-slave-9tm1 krb5kdc[23424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:30 dist-test-slave-9tm1 krb5kdc[23424](info): set up 2 sockets
Sep 24 12:51:30 dist-test-slave-9tm1 krb5kdc[23424](info): commencing operation
krb5kdc: starting...
W20260924 12:51:31.217543 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:31 dist-test-slave-9tm1 krb5kdc[23424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:31.482040 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:31.824352 23437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:31.824743 23437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:31.824892 23437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:31.834103 23437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:31.834251 23437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:31.834301 23437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:31.834362 23437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:31.834404 23437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:31.844606 23437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:31.845989 23437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:31.847270 23437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:31.859035 23442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:31.859620 23445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:31.859601 23443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:31.860248 23437 server_base.cc:1034] running on GCE node
I20260924 12:51:31.861106 23437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:31.862848 23437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:31.864082 23437 hybrid_clock.cc:648] HybridClock initialized: now 1790254291864036 us; error 64 us; skew 500 ppm
I20260924 12:51:31.864446 23437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:31 dist-test-slave-9tm1 krb5kdc[23424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:31.873212 23437 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:31.874873 23437 webserver.cc:460] Webserver started at http://127.0.0.1:46759/ using document root <none> and password file <none>
I20260924 12:51:31.875638 23437 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:31.875763 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:31.876055 23437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:31.879325 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "6df0719d548d44198841e5ca1c780af6"
format_stamp: "Formatted at 2026-09-24 12:51:31 on dist-test-slave-9tm1"
server_key: "e586f2456a6e0c74c424ad367e05ef96"
server_key_iv: "36c856640d128c89005d2ad64c5cca21"
server_key_version: "encryptionkey@0"
I20260924 12:51:31.880117 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "6df0719d548d44198841e5ca1c780af6"
format_stamp: "Formatted at 2026-09-24 12:51:31 on dist-test-slave-9tm1"
server_key: "e586f2456a6e0c74c424ad367e05ef96"
server_key_iv: "36c856640d128c89005d2ad64c5cca21"
server_key_version: "encryptionkey@0"
I20260924 12:51:31.886174 23437 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 12:51:31.890130 23452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:31.891867 23437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:51:31.892071 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "6df0719d548d44198841e5ca1c780af6"
format_stamp: "Formatted at 2026-09-24 12:51:31 on dist-test-slave-9tm1"
server_key: "e586f2456a6e0c74c424ad367e05ef96"
server_key_iv: "36c856640d128c89005d2ad64c5cca21"
server_key_version: "encryptionkey@0"
I20260924 12:51:31.892300 23437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:31.913542 23437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:31.918310 23437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:31.918601 23437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:31.937196 23437 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43031
I20260924 12:51:31.937189 23503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43031 every 8 connection(s)
I20260924 12:51:31.938611 23437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:31.942543 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:31.947165 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23437
I20260924 12:51:31.947348 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:31.947779 2373 external_mini_cluster.cc:1428] Setting key cfacd86f4044265eee0e871c542fc5bc
I20260924 12:51:31.955191 23504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6: Bootstrap starting.
I20260924 12:51:31.959673 23504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:31.960922 23504 log.cc:824] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6: Log is configured to *not* fsync() on all Append() calls
Sep 24 12:51:31 dist-test-slave-9tm1 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 12:51:31.964672 23504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6: No bootstrap required, opened a new log
I20260924 12:51:31.973626 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:31.949666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:37135 (local address 127.0.0.1:43031)
0924 12:51:31.950562 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:31.950577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:31.950624 (+ 47us) server_negotiation.cc:407] Connection header received
0924 12:51:31.952962 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:31.953052 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:31.953652 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:31.954116 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:31.955632 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:31.957084 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:31.958411 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:31.959163 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:31.959344 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:31.964370 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:31.964383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:31.964407 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:31.964474 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:31.968081 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:31.969233 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:31.969241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:31.969253 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:31.969480 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:31.970068 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:31.970077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:31.970079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:31.970850 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:31.971132 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:31.972011 (+ 879us) server_negotiation.cc:299] Negotiation successful
0924 12:51:31.972364 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":340,"threads_started":1}
I20260924 12:51:31.974818 23504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df0719d548d44198841e5ca1c780af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260924 12:51:31.975216 23504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:31.975319 23504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df0719d548d44198841e5ca1c780af6, State: Initialized, Role: FOLLOWER
I20260924 12:51:31.975939 23504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df0719d548d44198841e5ca1c780af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260924 12:51:31.976198 23504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:31.976373 23504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:31.976547 23504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:31.977890 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:31.979681 23504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df0719d548d44198841e5ca1c780af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260924 12:51:31.980386 23504 leader_election.cc:302] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df0719d548d44198841e5ca1c780af6; no voters:
I20260924 12:51:31.981175 23504 leader_election.cc:288] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:31.981292 23509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:31.982869 23509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [term 1 LEADER]: Becoming Leader. State: Replica: 6df0719d548d44198841e5ca1c780af6, State: Running, Role: LEADER
I20260924 12:51:31.983486 23509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df0719d548d44198841e5ca1c780af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } }
I20260924 12:51:31.983927 23504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:31.988564 23513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df0719d548d44198841e5ca1c780af6. Latest consensus state: current_term: 1 leader_uuid: "6df0719d548d44198841e5ca1c780af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df0719d548d44198841e5ca1c780af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } } }
I20260924 12:51:31.989001 23513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:31.994004 23512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df0719d548d44198841e5ca1c780af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df0719d548d44198841e5ca1c780af6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43031 } } }
I20260924 12:51:31.995039 23512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:32.006728 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:31.984645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:33063 (local address 127.0.0.1:43031)
0924 12:51:31.984820 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:31.984826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:31.984849 (+ 23us) server_negotiation.cc:407] Connection header received
0924 12:51:31.984912 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:31.984917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:31.985045 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:31.985174 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:31.987186 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:31.988841 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:31.991024 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:31.991828 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:31.991966 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:31.996273 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:31.996282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:31.996288 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:31.996347 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:32.000651 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:32.002353 (+ 1702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:32.002361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:32.002365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:32.002585 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:32.003434 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:32.003445 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:32.003448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:32.004021 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:32.004388 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:32.006029 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0924 12:51:32.006279 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:51:32.008378 23519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:32.014297 23519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:32.024322 23519 catalog_manager.cc:1347] Generated new cluster ID: 4de1eb3a64374a31b3be7e666890f521
I20260924 12:51:32.024454 23519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:32.041841 23519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:32.043958 23519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:32.063097 23519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df0719d548d44198841e5ca1c780af6: Generated new TSK 0
I20260924 12:51:32.064328 23519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:32.072312 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23437
I20260924 12:51:32.073614 23499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:32.169768 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23437
W20260924 12:51:32.185796 23422 connection.cc:535] client connection to 127.0.0.1:43031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43031): Connection reset by peer (error 104)
2026-09-24T12:51:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:32 dist-test-slave-9tm1 krb5kdc[23536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:32 dist-test-slave-9tm1 krb5kdc[23536](info): set up 2 sockets
Sep 24 12:51:32 dist-test-slave-9tm1 krb5kdc[23536](info): commencing operation
krb5kdc: starting...
W20260924 12:51:33.205663 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:33 dist-test-slave-9tm1 krb5kdc[23536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:33.485239 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:33.832779 23549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:33.833184 23549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:33.833299 23549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:33.841466 23549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:33.841616 23549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:33.841693 23549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:33.841774 23549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:33.841833 23549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:33.852870 23549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:33.854394 23549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:33.855808 23549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:33.873297 23554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:33.873273 23557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:33.873493 23555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:33.874240 23549 server_base.cc:1034] running on GCE node
I20260924 12:51:33.875110 23549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:33.876760 23549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:33.878021 23549 hybrid_clock.cc:648] HybridClock initialized: now 1790254293877967 us; error 55 us; skew 500 ppm
I20260924 12:51:33.878432 23549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:33 dist-test-slave-9tm1 krb5kdc[23536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:33.883308 23549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:33.885984 23549 webserver.cc:460] Webserver started at http://127.0.0.1:44337/ using document root <none> and password file <none>
I20260924 12:51:33.887190 23549 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:33.887380 23549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:33.887733 23549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:33.891316 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e"
format_stamp: "Formatted at 2026-09-24 12:51:33 on dist-test-slave-9tm1"
server_key: "5bd511bfccfb718c7e44d7392b4e3db2"
server_key_iv: "9927cf0409c5e09bbf7b42251eeaf083"
server_key_version: "encryptionkey@0"
I20260924 12:51:33.892287 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e"
format_stamp: "Formatted at 2026-09-24 12:51:33 on dist-test-slave-9tm1"
server_key: "5bd511bfccfb718c7e44d7392b4e3db2"
server_key_iv: "9927cf0409c5e09bbf7b42251eeaf083"
server_key_version: "encryptionkey@0"
I20260924 12:51:33.898409 23549 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 12:51:33.903249 23564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:33.904904 23549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:33.905153 23549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e"
format_stamp: "Formatted at 2026-09-24 12:51:33 on dist-test-slave-9tm1"
server_key: "5bd511bfccfb718c7e44d7392b4e3db2"
server_key_iv: "9927cf0409c5e09bbf7b42251eeaf083"
server_key_version: "encryptionkey@0"
I20260924 12:51:33.905361 23549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:33.927207 23549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:33.932190 23549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:33.932493 23549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:33.952571 23549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44497
I20260924 12:51:33.952591 23615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44497 every 8 connection(s)
I20260924 12:51:33.954237 23549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:33.958330 23616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:33.962342 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23549
I20260924 12:51:33.962591 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:33.963079 2373 external_mini_cluster.cc:1428] Setting key 71ff3b95e6d15ba6546efd1301641798
I20260924 12:51:33.973489 23616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e: Bootstrap starting.
I20260924 12:51:33.978492 23616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:33.978946 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:33.965090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:51003 (local address 127.0.0.1:44497)
0924 12:51:33.965958 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:33.965974 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:33.966023 (+ 49us) server_negotiation.cc:407] Connection header received
0924 12:51:33.968599 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:33.968635 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:33.969252 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:33.969728 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:33.970526 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:33.972013 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:33.973500 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:33.974274 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:33.974476 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:33.976301 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:33.976310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:33.976330 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:33.977008 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:33.977111 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:33.977155 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:33.977322 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:33.977632 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 12:51:33.977821 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":396,"threads_started":1}
I20260924 12:51:33.979889 23616 log.cc:824] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:33.983975 23616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e: No bootstrap required, opened a new log
I20260924 12:51:33.984386 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:33.994944 23616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260924 12:51:33.995435 23616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:33.996039 23616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc97f1dec9c4aed90f9e3a1e36da95e, State: Initialized, Role: FOLLOWER
I20260924 12:51:33.997006 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:33.989206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:36301 (local address 127.0.0.1:44497)
0924 12:51:33.989420 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:33.989426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:33.989676 (+ 250us) server_negotiation.cc:407] Connection header received
0924 12:51:33.989850 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:33.989855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:33.989959 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:33.990098 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:33.990587 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:33.991778 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:33.993483 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:33.994167 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:33.994303 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:33.996088 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:33.996095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:33.996100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:33.996126 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:33.996206 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:33.996231 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:33.996367 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:33.996585 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 12:51:33.996654 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 12:51:33.996929 23616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260924 12:51:33.997344 23616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:33.997502 23616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:33.997709 23616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:34.000491 23616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260924 12:51:34.001224 23616 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc97f1dec9c4aed90f9e3a1e36da95e; no voters:
I20260924 12:51:34.001906 23616 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:34.002035 23626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:34.002687 23626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [term 1 LEADER]: Becoming Leader. State: Replica: 1cc97f1dec9c4aed90f9e3a1e36da95e, State: Running, Role: LEADER
I20260924 12:51:34.003261 23626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } }
I20260924 12:51:34.003995 23616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:34.006400 23627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } } }
I20260924 12:51:34.006979 23627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:34.006943 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc97f1dec9c4aed90f9e3a1e36da95e. Latest consensus state: current_term: 1 leader_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc97f1dec9c4aed90f9e3a1e36da95e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44497 } } }
I20260924 12:51:34.009923 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:34.017776 23632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:34.023978 23632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:34.035048 23632 catalog_manager.cc:1347] Generated new cluster ID: cd0482c3828d4468a1261b8550acdaf3
I20260924 12:51:34.035302 23632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:34.046355 23632 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:34.048058 23632 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:34.060503 23632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc97f1dec9c4aed90f9e3a1e36da95e: Generated new TSK 0
I20260924 12:51:34.062556 23632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:34.071125 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23549
W20260924 12:51:34.072891 23611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:51003
W20260924 12:51:34.074620 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:34.076287 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23549
W20260924 12:51:34.094607 23534 connection.cc:535] client connection to 127.0.0.1:44497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44497): Connection reset by peer (error 104)
2026-09-24T12:51:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:34 dist-test-slave-9tm1 krb5kdc[23647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:34 dist-test-slave-9tm1 krb5kdc[23647](info): set up 2 sockets
Sep 24 12:51:34 dist-test-slave-9tm1 krb5kdc[23647](info): commencing operation
krb5kdc: starting...
W20260924 12:51:35.137537 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:35 dist-test-slave-9tm1 krb5kdc[23647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:35.398137 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:35.739075 23660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:35.739403 23660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:35.739542 23660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:35.747239 23660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:35.747402 23660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:35.747507 23660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:35.747592 23660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:35.747654 23660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:35.758298 23660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:35.759642 23660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:35.761096 23660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:35.775668 23665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:35.775645 23666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:35.776073 23668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:35.777220 23660 server_base.cc:1034] running on GCE node
I20260924 12:51:35.778203 23660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:35.779874 23660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:35.781157 23660 hybrid_clock.cc:648] HybridClock initialized: now 1790254295781078 us; error 72 us; skew 500 ppm
I20260924 12:51:35.781571 23660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:35 dist-test-slave-9tm1 krb5kdc[23647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:35.785794 23660 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:35.787658 23660 webserver.cc:460] Webserver started at http://127.0.0.1:37929/ using document root <none> and password file <none>
I20260924 12:51:35.788421 23660 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:35.788560 23660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:35.788865 23660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:35.793190 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5"
format_stamp: "Formatted at 2026-09-24 12:51:35 on dist-test-slave-9tm1"
server_key: "0b41f2bf139e30b3118670929ea34948"
server_key_iv: "2fe587619fcfe1a0fa6c845a7cf8711f"
server_key_version: "encryptionkey@0"
I20260924 12:51:35.794150 23660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5"
format_stamp: "Formatted at 2026-09-24 12:51:35 on dist-test-slave-9tm1"
server_key: "0b41f2bf139e30b3118670929ea34948"
server_key_iv: "2fe587619fcfe1a0fa6c845a7cf8711f"
server_key_version: "encryptionkey@0"
I20260924 12:51:35.801354 23660 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260924 12:51:35.805730 23675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:35.807525 23660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:51:35.807735 23660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5"
format_stamp: "Formatted at 2026-09-24 12:51:35 on dist-test-slave-9tm1"
server_key: "0b41f2bf139e30b3118670929ea34948"
server_key_iv: "2fe587619fcfe1a0fa6c845a7cf8711f"
server_key_version: "encryptionkey@0"
I20260924 12:51:35.807955 23660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:35.814782 23660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:35.819967 23660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:35.820303 23660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:35.839274 23660 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44397
I20260924 12:51:35.839313 23726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44397 every 8 connection(s)
I20260924 12:51:35.840649 23660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:35.844072 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23660
I20260924 12:51:35.844242 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:35.844550 23727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:35.844727 2373 external_mini_cluster.cc:1428] Setting key 216bd89539b41a993bac5ab8b4896362
I20260924 12:51:35.853405 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:35.846563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:47863 (local address 127.0.0.1:44397)
0924 12:51:35.847512 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:35.847525 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:35.847572 (+ 47us) server_negotiation.cc:407] Connection header received
0924 12:51:35.849740 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:35.849765 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:35.850198 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:35.850592 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:35.851703 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:35.851713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:35.851727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:35.851746 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:35.851809 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:35.851832 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:35.852063 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:35.852208 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 12:51:35.852386 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":458,"threads_started":1}
I20260924 12:51:35.858433 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:35.859385 23727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5: Bootstrap starting.
I20260924 12:51:35.862995 23727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:35.864545 23727 log.cc:824] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:35.865677 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:35.863134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:39253 (local address 127.0.0.1:44397)
0924 12:51:35.863318 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:35.863324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:35.863348 (+ 24us) server_negotiation.cc:407] Connection header received
0924 12:51:35.863502 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:35.863507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:35.863588 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:35.863727 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:35.864698 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:35.864703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:35.864708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:35.864726 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:35.864778 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:35.864799 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:35.864919 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:35.865257 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 12:51:35.865332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:51:35.867857 23727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5: No bootstrap required, opened a new log
I20260924 12:51:35.875511 23727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44397 } }
I20260924 12:51:35.875917 23727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:35.876026 23727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfb9e8a99f04eec83dc8cb76e0216c5, State: Initialized, Role: FOLLOWER
I20260924 12:51:35.876611 23727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44397 } }
I20260924 12:51:35.876832 23727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:35.876996 23727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:35.877166 23727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:35.879402 23727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44397 } }
I20260924 12:51:35.879908 23727 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfb9e8a99f04eec83dc8cb76e0216c5; no voters:
I20260924 12:51:35.880481 23727 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:35.880633 23737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:35.881681 23737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8cfb9e8a99f04eec83dc8cb76e0216c5, State: Running, Role: LEADER
I20260924 12:51:35.882258 23737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44397 } }
I20260924 12:51:35.882614 23727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:35.884852 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfb9e8a99f04eec83dc8cb76e0216c5. Latest consensus state: current_term: 1 leader_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44397 } } }
I20260924 12:51:35.885305 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:35.885133 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfb9e8a99f04eec83dc8cb76e0216c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44397 } } }
I20260924 12:51:35.885504 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:35.886234 23746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:35.892803 23746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:35.901836 23746 catalog_manager.cc:1347] Generated new cluster ID: c47817d2d1d34a8692ab8bead298153e
I20260924 12:51:35.901978 23746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:35.917478 23746 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:35.917932 23746 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:35.937523 23746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cfb9e8a99f04eec83dc8cb76e0216c5: Generated new TSK 0
I20260924 12:51:35.938526 23746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:35.954720 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23660
W20260924 12:51:35.956152 23722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:47863
W20260924 12:51:35.956877 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23660: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:35.957028 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23660
2026-09-24T12:51:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:36 dist-test-slave-9tm1 krb5kdc[23758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:36 dist-test-slave-9tm1 krb5kdc[23758](info): set up 2 sockets
Sep 24 12:51:36 dist-test-slave-9tm1 krb5kdc[23758](info): commencing operation
krb5kdc: starting...
W20260924 12:51:36.993454 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.011s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:37 dist-test-slave-9tm1 krb5kdc[23758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:37.258378 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:37.603520 23771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:37.603892 23771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:37.604027 23771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:37.611828 23771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:37.611968 23771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:37.612030 23771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:37.612111 23771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:37.612169 23771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:37.622447 23771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:37.623754 23771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:37.625119 23771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:37.636955 23776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:37.636927 23779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:37.636894 23777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:37.637295 23771 server_base.cc:1034] running on GCE node
I20260924 12:51:37.638382 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:37.639834 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:37.641083 23771 hybrid_clock.cc:648] HybridClock initialized: now 1790254297641015 us; error 52 us; skew 500 ppm
I20260924 12:51:37.641455 23771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:37 dist-test-slave-9tm1 krb5kdc[23758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:37.649755 23771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:37.651654 23771 webserver.cc:460] Webserver started at http://127.0.0.1:38139/ using document root <none> and password file <none>
I20260924 12:51:37.652431 23771 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:37.652560 23771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:37.652853 23771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:37.656522 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "bdf6ba99009a429aa0314dac5d26b29a"
format_stamp: "Formatted at 2026-09-24 12:51:37 on dist-test-slave-9tm1"
server_key: "34c18c1dbb2873305d48da6f2e136974"
server_key_iv: "4263331df5a9f23c4a56dbe88d605f33"
server_key_version: "encryptionkey@0"
I20260924 12:51:37.657739 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "bdf6ba99009a429aa0314dac5d26b29a"
format_stamp: "Formatted at 2026-09-24 12:51:37 on dist-test-slave-9tm1"
server_key: "34c18c1dbb2873305d48da6f2e136974"
server_key_iv: "4263331df5a9f23c4a56dbe88d605f33"
server_key_version: "encryptionkey@0"
I20260924 12:51:37.664316 23771 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 12:51:37.668517 23786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:37.670251 23771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:51:37.670459 23771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "bdf6ba99009a429aa0314dac5d26b29a"
format_stamp: "Formatted at 2026-09-24 12:51:37 on dist-test-slave-9tm1"
server_key: "34c18c1dbb2873305d48da6f2e136974"
server_key_iv: "4263331df5a9f23c4a56dbe88d605f33"
server_key_version: "encryptionkey@0"
I20260924 12:51:37.670713 23771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:37.693228 23771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:37.697738 23771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:37.698022 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:37.715994 23771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46015
I20260924 12:51:37.716017 23837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46015 every 8 connection(s)
I20260924 12:51:37.717368 23771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:37.721123 23838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:37.726233 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23771
I20260924 12:51:37.726408 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:37.726818 2373 external_mini_cluster.cc:1428] Setting key 1eeba6379102591a7762f0450439435e
I20260924 12:51:37.733349 23838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a: Bootstrap starting.
I20260924 12:51:37.737269 23838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:37.738413 23838 log.cc:824] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:37.741812 23838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a: No bootstrap required, opened a new log
Sep 24 12:51:37 dist-test-slave-9tm1 krb5kdc[23758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 12:51:37.751945 23838 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf6ba99009a429aa0314dac5d26b29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46015 } }
I20260924 12:51:37.752431 23838 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:37.752585 23838 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf6ba99009a429aa0314dac5d26b29a, State: Initialized, Role: FOLLOWER
I20260924 12:51:37.753335 23838 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf6ba99009a429aa0314dac5d26b29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46015 } }
I20260924 12:51:37.753652 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:37.728755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:49193 (local address 127.0.0.1:46015)
0924 12:51:37.729596 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:37.729613 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:37.729669 (+ 56us) server_negotiation.cc:407] Connection header received
0924 12:51:37.732096 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:37.732128 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:37.732688 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:37.733180 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:37.734614 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:37.736632 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:37.738121 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:37.739231 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:37.739394 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:37.743802 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:37.743814 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:37.743839 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:37.743888 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:37.748148 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:37.749298 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:37.749309 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:37.749324 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:37.749574 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:37.750102 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:37.750107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:37.750110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:37.750913 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:37.751249 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:37.751698 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 12:51:37.752063 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":347,"threads_started":1}
I20260924 12:51:37.753669 23838 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:37.753816 23838 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:37.754060 23838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:37.756848 23838 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf6ba99009a429aa0314dac5d26b29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46015 } }
I20260924 12:51:37.757409 23838 leader_election.cc:302] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf6ba99009a429aa0314dac5d26b29a; no voters:
I20260924 12:51:37.757653 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:37.758057 23838 leader_election.cc:288] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:37.758189 23843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:37.760036 23838 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:37.768172 23843 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [term 1 LEADER]: Becoming Leader. State: Replica: bdf6ba99009a429aa0314dac5d26b29a, State: Running, Role: LEADER
I20260924 12:51:37.769351 23843 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf6ba99009a429aa0314dac5d26b29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46015 } }
I20260924 12:51:37.772847 23844 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf6ba99009a429aa0314dac5d26b29a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf6ba99009a429aa0314dac5d26b29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46015 } } }
I20260924 12:51:37.773272 23844 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:37.772878 23845 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf6ba99009a429aa0314dac5d26b29a. Latest consensus state: current_term: 1 leader_uuid: "bdf6ba99009a429aa0314dac5d26b29a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf6ba99009a429aa0314dac5d26b29a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46015 } } }
I20260924 12:51:37.776407 23845 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a [sys.catalog]: This master's current role is: LEADER
W20260924 12:51:37.780270 23862 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:51:37.780416 23862 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:51:37.780619 23863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:37.783181 23863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:37.786056 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:37.769118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:42485 (local address 127.0.0.1:46015)
0924 12:51:37.769316 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:37.769323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:37.770201 (+ 878us) server_negotiation.cc:407] Connection header received
0924 12:51:37.770399 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:37.770406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:37.770507 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:37.770617 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:37.772402 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:37.773675 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:37.775204 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:37.775738 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:37.775853 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:37.778443 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:37.778448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:37.778451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:37.778483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:37.782221 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:37.783434 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:37.783440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:37.783444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:37.783605 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:37.784229 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:37.784235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:37.784239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:37.784693 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:37.785021 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:37.785470 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 12:51:37.785671 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 12:51:37.795329 23863 catalog_manager.cc:1347] Generated new cluster ID: b586bb7be92a4804927ce28f17e5e625
I20260924 12:51:37.795512 23863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:37.817756 23863 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:37.819700 23863 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:37.836898 23863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf6ba99009a429aa0314dac5d26b29a: Generated new TSK 0
I20260924 12:51:37.838416 23863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:37.851912 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23771
I20260924 12:51:37.853319 23833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:37.950407 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23771
W20260924 12:51:37.964587 23756 connection.cc:535] client connection to 127.0.0.1:46015 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46015): Connection reset by peer (error 104)
2026-09-24T12:51:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:38 dist-test-slave-9tm1 krb5kdc[23870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:38 dist-test-slave-9tm1 krb5kdc[23870](info): set up 2 sockets
Sep 24 12:51:38 dist-test-slave-9tm1 krb5kdc[23870](info): commencing operation
krb5kdc: starting...
W20260924 12:51:38.990173 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:39 dist-test-slave-9tm1 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:39.265980 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:39.625136 23883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:39.625437 23883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:39.625551 23883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:39.632946 23883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:39.633103 23883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:39.633176 23883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:39.633250 23883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:39.633303 23883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:39.644102 23883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:39.645696 23883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:39.647297 23883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:39.659456 23891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:39.659493 23889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:39.659956 23888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:39.660547 23883 server_base.cc:1034] running on GCE node
I20260924 12:51:39.661412 23883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:39.662930 23883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:39.664150 23883 hybrid_clock.cc:648] HybridClock initialized: now 1790254299664097 us; error 53 us; skew 500 ppm
I20260924 12:51:39.664544 23883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:39 dist-test-slave-9tm1 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:39.667824 23883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:39.669404 23883 webserver.cc:460] Webserver started at http://127.0.0.1:43591/ using document root <none> and password file <none>
I20260924 12:51:39.670109 23883 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:39.670229 23883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:39.670526 23883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:39.673758 23883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "ac6539f7b67a47f28b649e62d9a68855"
format_stamp: "Formatted at 2026-09-24 12:51:39 on dist-test-slave-9tm1"
server_key: "120e1b0259910667d48436aaf1358033"
server_key_iv: "546b68c5d7f8bfbabe85f42237c7a3fe"
server_key_version: "encryptionkey@0"
I20260924 12:51:39.674588 23883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "ac6539f7b67a47f28b649e62d9a68855"
format_stamp: "Formatted at 2026-09-24 12:51:39 on dist-test-slave-9tm1"
server_key: "120e1b0259910667d48436aaf1358033"
server_key_iv: "546b68c5d7f8bfbabe85f42237c7a3fe"
server_key_version: "encryptionkey@0"
I20260924 12:51:39.680198 23883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:51:39.684190 23898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:39.685788 23883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 12:51:39.685974 23883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "ac6539f7b67a47f28b649e62d9a68855"
format_stamp: "Formatted at 2026-09-24 12:51:39 on dist-test-slave-9tm1"
server_key: "120e1b0259910667d48436aaf1358033"
server_key_iv: "546b68c5d7f8bfbabe85f42237c7a3fe"
server_key_version: "encryptionkey@0"
I20260924 12:51:39.686187 23883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:39.708302 23883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:39.713231 23883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:39.713537 23883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:39.731992 23883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35923
I20260924 12:51:39.732030 23949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35923 every 8 connection(s)
I20260924 12:51:39.733429 23883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:39.737254 23950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:39.742129 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23883
I20260924 12:51:39.742328 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:39.742861 2373 external_mini_cluster.cc:1428] Setting key 3824312873bb2c4dfeae1c80db1faa19
I20260924 12:51:39.749035 23950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855: Bootstrap starting.
I20260924 12:51:39.752725 23950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:39.754228 23950 log.cc:824] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:39.757551 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:39.744847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:33389 (local address 127.0.0.1:35923)
0924 12:51:39.745728 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:39.745740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:39.745786 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:51:39.747600 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:39.747626 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:39.748067 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:39.748416 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:39.749064 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:39.750457 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:39.752335 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:39.753169 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:39.753322 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:39.754845 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:39.754857 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:39.754879 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:39.755707 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:39.755808 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:39.755841 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:39.755968 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:39.756334 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 12:51:39.756544 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":402,"threads_started":1}
I20260924 12:51:39.757643 23950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855: No bootstrap required, opened a new log
I20260924 12:51:39.761523 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:39.767053 23950 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6539f7b67a47f28b649e62d9a68855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260924 12:51:39.767588 23950 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:39.767711 23950 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6539f7b67a47f28b649e62d9a68855, State: Initialized, Role: FOLLOWER
I20260924 12:51:39.768419 23950 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6539f7b67a47f28b649e62d9a68855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260924 12:51:39.768679 23950 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:39.768823 23950 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:39.769007 23950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:39.772107 23950 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6539f7b67a47f28b649e62d9a68855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260924 12:51:39.772737 23950 leader_election.cc:302] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6539f7b67a47f28b649e62d9a68855; no voters:
I20260924 12:51:39.773442 23950 leader_election.cc:288] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:39.773620 23960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:39.774236 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:39.766863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:53683 (local address 127.0.0.1:35923)
0924 12:51:39.767118 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:39.767124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:39.767361 (+ 237us) server_negotiation.cc:407] Connection header received
0924 12:51:39.767626 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:39.767631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:39.767719 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:39.767815 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:39.768367 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:39.769402 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:39.771265 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:39.771876 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:39.771990 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:39.773227 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:39.773238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:39.773243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:39.773270 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:39.773327 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:39.773350 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:39.773558 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:39.773808 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 12:51:39.773877 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 12:51:39.776257 23950 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:39.780951 23960 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [term 1 LEADER]: Becoming Leader. State: Replica: ac6539f7b67a47f28b649e62d9a68855, State: Running, Role: LEADER
I20260924 12:51:39.781569 23960 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6539f7b67a47f28b649e62d9a68855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } }
I20260924 12:51:39.784354 23962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6539f7b67a47f28b649e62d9a68855. Latest consensus state: current_term: 1 leader_uuid: "ac6539f7b67a47f28b649e62d9a68855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6539f7b67a47f28b649e62d9a68855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } } }
I20260924 12:51:39.784596 23961 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6539f7b67a47f28b649e62d9a68855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6539f7b67a47f28b649e62d9a68855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35923 } } }
I20260924 12:51:39.785269 23961 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:39.785274 23962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:39.786319 23970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:39.791903 23970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:39.800798 23970 catalog_manager.cc:1347] Generated new cluster ID: 284a3fc78d1b41709e7816a954b97265
I20260924 12:51:39.801021 23970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:39.819517 23970 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:39.820817 23970 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:39.830273 23970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6539f7b67a47f28b649e62d9a68855: Generated new TSK 0
I20260924 12:51:39.831110 23970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:39.843410 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23883
W20260924 12:51:39.845213 23945 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:33389
W20260924 12:51:39.846093 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23883: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:39.846239 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23883
W20260924 12:51:39.859666 23868 connection.cc:535] client connection to 127.0.0.1:35923 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35923): Connection reset by peer (error 104)
2026-09-24T12:51:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:39 dist-test-slave-9tm1 krb5kdc[23981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:39 dist-test-slave-9tm1 krb5kdc[23981](info): set up 2 sockets
Sep 24 12:51:39 dist-test-slave-9tm1 krb5kdc[23981](info): commencing operation
krb5kdc: starting...
W20260924 12:51:40.881392 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:41 dist-test-slave-9tm1 krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 12:51:41.137724 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:41.484951 23994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:41.485284 23994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:41.485416 23994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:41.492733 23994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:41.492870 23994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:41.492926 23994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:41.493034 23994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:41.493089 23994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:41.503737 23994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:41.505098 23994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:41.506356 23994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:41.518535 24002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:41.518556 23999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:41.518498 24000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:41.518968 23994 server_base.cc:1034] running on GCE node
I20260924 12:51:41.519927 23994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:41.521453 23994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:41.522694 23994 hybrid_clock.cc:648] HybridClock initialized: now 1790254301522650 us; error 33 us; skew 500 ppm
I20260924 12:51:41.523078 23994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:41 dist-test-slave-9tm1 krb5kdc[23981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:41.526363 23994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 12:51:41.527899 23994 webserver.cc:460] Webserver started at http://127.0.0.1:45583/ using document root <none> and password file <none>
I20260924 12:51:41.528591 23994 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:41.528713 23994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:41.529042 23994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:41.532148 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "70596d16ea8b49638d760fd21fe3bb67"
format_stamp: "Formatted at 2026-09-24 12:51:41 on dist-test-slave-9tm1"
server_key: "ccc2b00ce6fcdbf1195d0fe263e9f7b5"
server_key_iv: "01507751e533b93ecddb89d6a8ab3c43"
server_key_version: "encryptionkey@0"
I20260924 12:51:41.533072 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "70596d16ea8b49638d760fd21fe3bb67"
format_stamp: "Formatted at 2026-09-24 12:51:41 on dist-test-slave-9tm1"
server_key: "ccc2b00ce6fcdbf1195d0fe263e9f7b5"
server_key_iv: "01507751e533b93ecddb89d6a8ab3c43"
server_key_version: "encryptionkey@0"
I20260924 12:51:41.538750 23994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 12:51:41.542690 24009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:41.544193 23994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:51:41.544385 23994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "70596d16ea8b49638d760fd21fe3bb67"
format_stamp: "Formatted at 2026-09-24 12:51:41 on dist-test-slave-9tm1"
server_key: "ccc2b00ce6fcdbf1195d0fe263e9f7b5"
server_key_iv: "01507751e533b93ecddb89d6a8ab3c43"
server_key_version: "encryptionkey@0"
I20260924 12:51:41.544634 23994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:41.551101 23994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:41.556015 23994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:41.556356 23994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:41.575003 23994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42949
I20260924 12:51:41.575038 24060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42949 every 8 connection(s)
I20260924 12:51:41.576524 23994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:41.580480 24061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:41.583036 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 23994
I20260924 12:51:41.583213 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:41.583719 2373 external_mini_cluster.cc:1428] Setting key e6e89a26ccd6f1db337725c849c3dd9f
I20260924 12:51:41.591712 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:41.585693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:59907 (local address 127.0.0.1:42949)
0924 12:51:41.586559 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:41.586571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:41.586613 (+ 42us) server_negotiation.cc:407] Connection header received
0924 12:51:41.588239 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:41.588261 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:41.588660 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:41.589026 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:41.590069 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:41.590075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:41.590089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:41.590104 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:41.590175 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:41.590200 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:41.590380 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:41.590488 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 12:51:41.590768 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":402,"threads_started":1}
I20260924 12:51:41.592816 24061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67: Bootstrap starting.
I20260924 12:51:41.595517 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:41.596405 24061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:41.597689 24061 log.cc:824] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:41.600864 24061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67: No bootstrap required, opened a new log
I20260924 12:51:41.602847 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:41.600321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:33783 (local address 127.0.0.1:42949)
0924 12:51:41.600460 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:41.600465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:41.600997 (+ 532us) server_negotiation.cc:407] Connection header received
0924 12:51:41.601164 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:41.601170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:41.601220 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:41.601301 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:41.602235 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:41.602240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:41.602245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:41.602266 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:41.602303 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:41.602318 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:41.602422 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:41.602505 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 12:51:41.602572 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 12:51:41.609012 24061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70596d16ea8b49638d760fd21fe3bb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 12:51:41.609424 24061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:41.609527 24061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70596d16ea8b49638d760fd21fe3bb67, State: Initialized, Role: FOLLOWER
I20260924 12:51:41.610121 24061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70596d16ea8b49638d760fd21fe3bb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 12:51:41.610399 24061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:41.610546 24061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:41.610700 24061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:41.612874 24061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70596d16ea8b49638d760fd21fe3bb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 12:51:41.613402 24061 leader_election.cc:302] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70596d16ea8b49638d760fd21fe3bb67; no voters:
I20260924 12:51:41.614019 24061 leader_election.cc:288] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:41.614159 24071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:41.614801 24071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [term 1 LEADER]: Becoming Leader. State: Replica: 70596d16ea8b49638d760fd21fe3bb67, State: Running, Role: LEADER
I20260924 12:51:41.615320 24071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70596d16ea8b49638d760fd21fe3bb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } }
I20260924 12:51:41.616001 24061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:41.618783 24073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70596d16ea8b49638d760fd21fe3bb67. Latest consensus state: current_term: 1 leader_uuid: "70596d16ea8b49638d760fd21fe3bb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70596d16ea8b49638d760fd21fe3bb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } } }
I20260924 12:51:41.618737 24072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70596d16ea8b49638d760fd21fe3bb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70596d16ea8b49638d760fd21fe3bb67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42949 } } }
I20260924 12:51:41.619249 24072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:41.619249 24073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:41.620110 24081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:41.626893 24081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:41.637607 24081 catalog_manager.cc:1347] Generated new cluster ID: f4d0db62cc334084a68e93be3ef72db1
I20260924 12:51:41.637912 24081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:41.657111 24081 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:41.657778 24081 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:41.682154 24081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70596d16ea8b49638d760fd21fe3bb67: Generated new TSK 0
I20260924 12:51:41.683290 24081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:41.698036 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23994
W20260924 12:51:41.699523 24056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.154:59907
W20260924 12:51:41.700266 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 23994: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:41.700428 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 23994
2026-09-24T12:51:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:41 dist-test-slave-9tm1 krb5kdc[24092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:41 dist-test-slave-9tm1 krb5kdc[24092](info): set up 2 sockets
Sep 24 12:51:41 dist-test-slave-9tm1 krb5kdc[24092](info): commencing operation
krb5kdc: starting...
W20260924 12:51:42.737481 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:42 dist-test-slave-9tm1 krb5kdc[24092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:43.021010 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36321
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:43.356062 24105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:43.356387 24105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:43.356511 24105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:43.364045 24105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:43.364197 24105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:43.364276 24105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:43.364355 24105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:43.364416 24105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:43.375350 24105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:36321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:43.376693 24105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:43.378069 24105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:43.392661 24111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:43.392688 24113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:43.395217 24110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:43.397786 24105 server_base.cc:1034] running on GCE node
I20260924 12:51:43.398890 24105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:43.401039 24105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:43.402326 24105 hybrid_clock.cc:648] HybridClock initialized: now 1790254303402246 us; error 75 us; skew 500 ppm
I20260924 12:51:43.402823 24105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:43 dist-test-slave-9tm1 krb5kdc[24092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254303, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:43.407341 24105 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:43.409343 24105 webserver.cc:460] Webserver started at http://127.2.81.126:36683/ using document root <none> and password file <none>
I20260924 12:51:43.410354 24105 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:43.410617 24105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:43.411170 24105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:43.415966 24105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "adbfbdfb3a1840ae967eef06363a2773"
format_stamp: "Formatted at 2026-09-24 12:51:43 on dist-test-slave-9tm1"
server_key: "bc9324a1810966fbbcde37016d1467ed"
server_key_iv: "a4a81352248de33ee0847f995072bd5d"
server_key_version: "encryptionkey@0"
I20260924 12:51:43.417420 24105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "adbfbdfb3a1840ae967eef06363a2773"
format_stamp: "Formatted at 2026-09-24 12:51:43 on dist-test-slave-9tm1"
server_key: "bc9324a1810966fbbcde37016d1467ed"
server_key_iv: "a4a81352248de33ee0847f995072bd5d"
server_key_version: "encryptionkey@0"
I20260924 12:51:43.425808 24105 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260924 12:51:43.429955 24120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:43.431572 24105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:43.431855 24105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "adbfbdfb3a1840ae967eef06363a2773"
format_stamp: "Formatted at 2026-09-24 12:51:43 on dist-test-slave-9tm1"
server_key: "bc9324a1810966fbbcde37016d1467ed"
server_key_iv: "a4a81352248de33ee0847f995072bd5d"
server_key_version: "encryptionkey@0"
I20260924 12:51:43.432109 24105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:43.447132 24105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:43.451807 24105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:43.452090 24105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:43.470302 24105 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:36321
I20260924 12:51:43.470325 24171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:36321 every 8 connection(s)
I20260924 12:51:43.471776 24105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:43.475718 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:43.476818 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24105
I20260924 12:51:43.477016 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:43.477553 2373 external_mini_cluster.cc:1428] Setting key 96b90e8bab234cd196f41d2b473e4dc7
I20260924 12:51:43.488938 24172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773: Bootstrap starting.
Sep 24 12:51:43 dist-test-slave-9tm1 krb5kdc[24092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:43.492286 24172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:43.493479 24172 log.cc:824] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:43.496371 24172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773: No bootstrap required, opened a new log
I20260924 12:51:43.499928 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:43.479461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36765 (local address 127.2.81.126:36321)
0924 12:51:43.480256 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:43.480270 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:43.480313 (+ 43us) server_negotiation.cc:407] Connection header received
0924 12:51:43.482259 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:43.482289 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:43.482791 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:43.483189 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:43.484674 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:43.486100 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:43.487419 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:43.488150 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:43.488330 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:43.492645 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:43.492657 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:43.492672 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:43.492724 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:43.495754 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:43.496542 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:43.496549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:43.496555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:43.496726 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:43.497228 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:43.497232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:43.497234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:43.497760 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:43.498086 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:43.498555 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 12:51:43.498898 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":566,"thread_start_us":337,"threads_started":1}
I20260924 12:51:43.504043 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:43.505057 24172 raft_consensus.cc:348] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfbdfb3a1840ae967eef06363a2773" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36321 } }
I20260924 12:51:43.505417 24172 raft_consensus.cc:374] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:43.505514 24172 raft_consensus.cc:729] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbfbdfb3a1840ae967eef06363a2773, State: Initialized, Role: FOLLOWER
I20260924 12:51:43.506062 24172 consensus_queue.cc:260] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfbdfb3a1840ae967eef06363a2773" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36321 } }
I20260924 12:51:43.506268 24172 raft_consensus.cc:388] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:43.506393 24172 raft_consensus.cc:482] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:43.506542 24172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:43.509186 24172 raft_consensus.cc:504] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfbdfb3a1840ae967eef06363a2773" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36321 } }
I20260924 12:51:43.509663 24172 leader_election.cc:302] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adbfbdfb3a1840ae967eef06363a2773; no voters:
I20260924 12:51:43.510241 24172 leader_election.cc:288] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:43.510362 24182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:43.511464 24182 raft_consensus.cc:686] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [term 1 LEADER]: Becoming Leader. State: Replica: adbfbdfb3a1840ae967eef06363a2773, State: Running, Role: LEADER
I20260924 12:51:43.512027 24182 consensus_queue.cc:237] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfbdfb3a1840ae967eef06363a2773" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36321 } }
I20260924 12:51:43.512605 24172 sys_catalog.cc:564] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:43.514799 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adbfbdfb3a1840ae967eef06363a2773. Latest consensus state: current_term: 1 leader_uuid: "adbfbdfb3a1840ae967eef06363a2773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfbdfb3a1840ae967eef06363a2773" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36321 } } }
I20260924 12:51:43.514945 24183 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adbfbdfb3a1840ae967eef06363a2773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbfbdfb3a1840ae967eef06363a2773" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36321 } } }
I20260924 12:51:43.515250 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:43.515282 24183 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:43.516099 24187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:43.525384 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:43.508231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46893 (local address 127.2.81.126:36321)
0924 12:51:43.508376 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:43.508380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:43.508854 (+ 474us) server_negotiation.cc:407] Connection header received
0924 12:51:43.509047 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:43.509053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:43.509113 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:43.509207 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:43.510600 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:43.511788 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:43.513192 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:43.513759 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:43.514546 (+ 787us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:43.517111 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:43.517119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:43.517123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:43.517170 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:43.520911 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:43.522049 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:43.522055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:43.522060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:43.522258 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:43.522822 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:43.522827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:43.522831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:43.523270 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:43.524512 (+ 1242us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:43.524700 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 12:51:43.524912 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 12:51:43.529191 24187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:43.540043 24187 catalog_manager.cc:1347] Generated new cluster ID: b6eb6bfa1a1644e385573fce02ddacc9
I20260924 12:51:43.540201 24187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:43.553606 24187 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:43.555526 24187 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:43.570537 24187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adbfbdfb3a1840ae967eef06363a2773: Generated new TSK 0
I20260924 12:51:43.571460 24187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:43.581807 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24105
I20260924 12:51:43.583511 24167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:43.675148 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24105
W20260924 12:51:43.688653 24090 connection.cc:535] client connection to 127.2.81.126:36321 recv error: Network error: failed to read from TLS socket (remote: 127.2.81.126:36321): Connection reset by peer (error 104)
2026-09-24T12:51:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:43 dist-test-slave-9tm1 krb5kdc[24204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:43 dist-test-slave-9tm1 krb5kdc[24204](info): set up 2 sockets
Sep 24 12:51:43 dist-test-slave-9tm1 krb5kdc[24204](info): commencing operation
krb5kdc: starting...
W20260924 12:51:44.705302 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:44 dist-test-slave-9tm1 krb5kdc[24204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:44.957234 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36155
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:45.303475 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:45.303797 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:45.303925 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:45.312107 24217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:45.312248 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:45.312326 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:45.312412 24217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:45.312484 24217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:45.322897 24217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:36155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:45.324273 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:45.325666 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:45.338186 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:45.338150 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:45.338146 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:45.339052 24217 server_base.cc:1034] running on GCE node
I20260924 12:51:45.339864 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:45.341459 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:45.342751 24217 hybrid_clock.cc:648] HybridClock initialized: now 1790254305342705 us; error 35 us; skew 500 ppm
I20260924 12:51:45.343166 24217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:45 dist-test-slave-9tm1 krb5kdc[24204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254305, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:45.346644 24217 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:45.348363 24217 webserver.cc:460] Webserver started at http://127.2.81.126:44127/ using document root <none> and password file <none>
I20260924 12:51:45.349176 24217 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:45.349309 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:45.349622 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:45.352773 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "05d9d17a16d94f0ebd55c836b5d455ed"
format_stamp: "Formatted at 2026-09-24 12:51:45 on dist-test-slave-9tm1"
server_key: "544188a1c205f5b30fe5516a8112b27d"
server_key_iv: "cd97a69d42198a70b41f93a061a5b173"
server_key_version: "encryptionkey@0"
I20260924 12:51:45.353649 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "05d9d17a16d94f0ebd55c836b5d455ed"
format_stamp: "Formatted at 2026-09-24 12:51:45 on dist-test-slave-9tm1"
server_key: "544188a1c205f5b30fe5516a8112b27d"
server_key_iv: "cd97a69d42198a70b41f93a061a5b173"
server_key_version: "encryptionkey@0"
I20260924 12:51:45.359169 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 12:51:45.363127 24232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:45.364573 24217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 12:51:45.364754 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "05d9d17a16d94f0ebd55c836b5d455ed"
format_stamp: "Formatted at 2026-09-24 12:51:45 on dist-test-slave-9tm1"
server_key: "544188a1c205f5b30fe5516a8112b27d"
server_key_iv: "cd97a69d42198a70b41f93a061a5b173"
server_key_version: "encryptionkey@0"
I20260924 12:51:45.364949 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:45.392086 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:45.396896 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:45.397233 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:45.416231 24217 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:36155
I20260924 12:51:45.416252 24283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:36155 every 8 connection(s)
I20260924 12:51:45.417729 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:45.421633 24284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:45.423406 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24217
I20260924 12:51:45.423594 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:45.424132 2373 external_mini_cluster.cc:1428] Setting key 7e6ba28be82fdf9925cf7b40ab389857
I20260924 12:51:45.436558 24284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed: Bootstrap starting.
I20260924 12:51:45.439272 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:45.426098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44175 (local address 127.2.81.126:36155)
0924 12:51:45.426967 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:45.426980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:45.427025 (+ 45us) server_negotiation.cc:407] Connection header received
0924 12:51:45.429047 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:45.429079 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:45.429541 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:45.429901 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:45.430615 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:45.432116 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:45.434203 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:45.435033 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:45.435183 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:45.436503 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:45.436511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:45.436527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:45.437331 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:45.437423 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:45.437457 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:45.437605 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:45.437987 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 12:51:45.438208 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":380,"threads_started":1}
I20260924 12:51:45.440132 24284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:45.441376 24284 log.cc:824] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:45.443773 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:45.444495 24284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed: No bootstrap required, opened a new log
I20260924 12:51:45.454581 24284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36155 } }
I20260924 12:51:45.455094 24284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:45.455245 24284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d9d17a16d94f0ebd55c836b5d455ed, State: Initialized, Role: FOLLOWER
I20260924 12:51:45.455770 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:45.449000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38307 (local address 127.2.81.126:36155)
0924 12:51:45.449195 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:45.449201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:45.449539 (+ 338us) server_negotiation.cc:407] Connection header received
0924 12:51:45.449760 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:45.449767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:45.449842 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:45.449934 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:45.450542 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:45.451332 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:45.453182 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:45.453671 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:45.453786 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:45.454979 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:45.454984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:45.454988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:45.455010 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:45.455062 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:45.455088 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:45.455222 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:45.455421 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 12:51:45.455478 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 12:51:45.456040 24284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36155 } }
I20260924 12:51:45.456331 24284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:45.456506 24284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:45.456676 24284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:45.459190 24284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36155 } }
I20260924 12:51:45.459721 24284 leader_election.cc:302] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d9d17a16d94f0ebd55c836b5d455ed; no voters:
I20260924 12:51:45.460373 24284 leader_election.cc:288] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:45.460485 24294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:45.461771 24294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [term 1 LEADER]: Becoming Leader. State: Replica: 05d9d17a16d94f0ebd55c836b5d455ed, State: Running, Role: LEADER
I20260924 12:51:45.462335 24294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36155 } }
I20260924 12:51:45.462706 24284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:45.465085 24295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36155 } } }
I20260924 12:51:45.465191 24296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d9d17a16d94f0ebd55c836b5d455ed. Latest consensus state: current_term: 1 leader_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d9d17a16d94f0ebd55c836b5d455ed" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36155 } } }
I20260924 12:51:45.465620 24296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:45.465598 24295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:45.466558 24302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:45.475337 24302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:45.486280 24302 catalog_manager.cc:1347] Generated new cluster ID: 931513c4c2594415a44c0450d3e766c5
I20260924 12:51:45.486426 24302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:45.501689 24302 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:45.503218 24302 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:45.510691 24302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d9d17a16d94f0ebd55c836b5d455ed: Generated new TSK 0
I20260924 12:51:45.511656 24302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:45.519052 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24217
W20260924 12:51:45.520471 24279 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44175
W20260924 12:51:45.521358 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:45.521502 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24217
W20260924 12:51:45.535017 24202 connection.cc:535] client connection to 127.2.81.126:36155 recv error: Network error: failed to read from TLS socket (remote: 127.2.81.126:36155): Connection reset by peer (error 104)
2026-09-24T12:51:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:45 dist-test-slave-9tm1 krb5kdc[24315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:45 dist-test-slave-9tm1 krb5kdc[24315](info): set up 2 sockets
Sep 24 12:51:45 dist-test-slave-9tm1 krb5kdc[24315](info): commencing operation
krb5kdc: starting...
W20260924 12:51:46.561504 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:46 dist-test-slave-9tm1 krb5kdc[24315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:46.832630 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44461
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:47.167798 24328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:47.168116 24328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:47.168262 24328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:47.175813 24328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:47.175936 24328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:47.176011 24328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:47.176088 24328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:47.176131 24328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:47.186913 24328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:44461
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:47.188539 24328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:47.190290 24328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:47.203809 24336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:47.203788 24333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:47.203855 24334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:47.204232 24328 server_base.cc:1034] running on GCE node
I20260924 12:51:47.205263 24328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:47.207218 24328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:47.208475 24328 hybrid_clock.cc:648] HybridClock initialized: now 1790254307208429 us; error 39 us; skew 500 ppm
I20260924 12:51:47.208921 24328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:47 dist-test-slave-9tm1 krb5kdc[24315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254307, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:47.212467 24328 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:47.214273 24328 webserver.cc:460] Webserver started at http://127.2.81.126:33305/ using document root <none> and password file <none>
I20260924 12:51:47.215003 24328 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:47.215127 24328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:47.215422 24328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:47.218652 24328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "2af97f4c5c1f44219c9ac025f7bb5e70"
format_stamp: "Formatted at 2026-09-24 12:51:47 on dist-test-slave-9tm1"
server_key: "cc5e25079cc97cd655332db29a27b1ee"
server_key_iv: "0e1282030c40f156888754b1d5fdfe07"
server_key_version: "encryptionkey@0"
I20260924 12:51:47.219511 24328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "2af97f4c5c1f44219c9ac025f7bb5e70"
format_stamp: "Formatted at 2026-09-24 12:51:47 on dist-test-slave-9tm1"
server_key: "cc5e25079cc97cd655332db29a27b1ee"
server_key_iv: "0e1282030c40f156888754b1d5fdfe07"
server_key_version: "encryptionkey@0"
I20260924 12:51:47.225637 24328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 12:51:47.229512 24343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:47.231324 24328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:47.231524 24328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "2af97f4c5c1f44219c9ac025f7bb5e70"
format_stamp: "Formatted at 2026-09-24 12:51:47 on dist-test-slave-9tm1"
server_key: "cc5e25079cc97cd655332db29a27b1ee"
server_key_iv: "0e1282030c40f156888754b1d5fdfe07"
server_key_version: "encryptionkey@0"
I20260924 12:51:47.231972 24328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:47.238263 24328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:47.243052 24328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:47.243388 24328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:47.262362 24328 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:44461
I20260924 12:51:47.262405 24394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:44461 every 8 connection(s)
I20260924 12:51:47.263777 24328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:47.267424 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24328
I20260924 12:51:47.267568 24395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:47.267621 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:47.268155 2373 external_mini_cluster.cc:1428] Setting key e6740f2db6e356fc7f190798b00d9bc4
I20260924 12:51:47.276189 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:47.269993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55317 (local address 127.2.81.126:44461)
0924 12:51:47.270904 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:47.270917 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:47.270963 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:51:47.272563 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:47.272588 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:47.273042 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:47.273421 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:47.274552 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:47.274559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:47.274573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:47.274588 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:47.274692 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:47.274722 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:47.274910 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:47.275015 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 12:51:47.275224 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":442,"threads_started":1}
I20260924 12:51:47.279994 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:47.281870 24395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70: Bootstrap starting.
I20260924 12:51:47.286339 24395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:47.287768 24395 log.cc:824] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:47.288033 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:47.285324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39993 (local address 127.2.81.126:44461)
0924 12:51:47.285479 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:47.285484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:47.285895 (+ 411us) server_negotiation.cc:407] Connection header received
0924 12:51:47.286053 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:47.286057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:47.286112 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:47.286207 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:47.287150 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:47.287155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:47.287158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:47.287175 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:47.287218 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:47.287234 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:47.287337 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:47.287571 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 12:51:47.287648 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 12:51:47.290802 24395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70: No bootstrap required, opened a new log
I20260924 12:51:47.298256 24395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44461 } }
I20260924 12:51:47.298652 24395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:47.298751 24395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af97f4c5c1f44219c9ac025f7bb5e70, State: Initialized, Role: FOLLOWER
I20260924 12:51:47.299327 24395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44461 } }
I20260924 12:51:47.299548 24395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:47.299678 24395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:47.299814 24395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:47.302141 24395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44461 } }
I20260924 12:51:47.302645 24395 leader_election.cc:302] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2af97f4c5c1f44219c9ac025f7bb5e70; no voters:
I20260924 12:51:47.303223 24395 leader_election.cc:288] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:47.303368 24405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:47.304026 24405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [term 1 LEADER]: Becoming Leader. State: Replica: 2af97f4c5c1f44219c9ac025f7bb5e70, State: Running, Role: LEADER
I20260924 12:51:47.304576 24405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44461 } }
I20260924 12:51:47.305238 24395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:47.307329 24406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44461 } } }
I20260924 12:51:47.308248 24406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:47.307839 24407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2af97f4c5c1f44219c9ac025f7bb5e70. Latest consensus state: current_term: 1 leader_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af97f4c5c1f44219c9ac025f7bb5e70" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 44461 } } }
I20260924 12:51:47.308830 24407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:47.312804 24411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:47.317920 24411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:47.327649 24411 catalog_manager.cc:1347] Generated new cluster ID: 95f2313cb40a4cf690ad2d133a466f59
I20260924 12:51:47.327780 24411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:47.346393 24411 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:47.346943 24411 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:47.356408 24411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2af97f4c5c1f44219c9ac025f7bb5e70: Generated new TSK 0
I20260924 12:51:47.357324 24411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:47.361655 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24328
W20260924 12:51:47.362839 24390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55317
W20260924 12:51:47.363586 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:47.363719 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24328
2026-09-24T12:51:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:47 dist-test-slave-9tm1 krb5kdc[24426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:47 dist-test-slave-9tm1 krb5kdc[24426](info): set up 2 sockets
Sep 24 12:51:47 dist-test-slave-9tm1 krb5kdc[24426](info): commencing operation
krb5kdc: starting...
W20260924 12:51:48.389576 2373 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:48 dist-test-slave-9tm1 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:48.665390 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39647
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:49.013610 24439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:49.013942 24439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:49.014083 24439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:49.021752 24439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:49.021903 24439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:49.021996 24439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:49.022177 24439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:49.022286 24439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:49.032532 24439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:49.033866 24439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:49.035176 24439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:49.047041 24445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:49.047065 24444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:49.047024 24447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:49.048210 24439 server_base.cc:1034] running on GCE node
I20260924 12:51:49.049185 24439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:49.050786 24439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:49.052057 24439 hybrid_clock.cc:648] HybridClock initialized: now 1790254309051997 us; error 52 us; skew 500 ppm
I20260924 12:51:49.052425 24439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:49 dist-test-slave-9tm1 krb5kdc[24426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:49.062857 24439 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:49.064476 24439 webserver.cc:460] Webserver started at http://127.2.81.126:36757/ using document root <none> and password file <none>
I20260924 12:51:49.065234 24439 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:49.065346 24439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:49.065596 24439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:49.068629 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d"
format_stamp: "Formatted at 2026-09-24 12:51:49 on dist-test-slave-9tm1"
server_key: "0fa276a9884513eb9467fc09030e5f44"
server_key_iv: "3e6c9cc2abc50486d93ad0ba0878f6c5"
server_key_version: "encryptionkey@0"
I20260924 12:51:49.069494 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d"
format_stamp: "Formatted at 2026-09-24 12:51:49 on dist-test-slave-9tm1"
server_key: "0fa276a9884513eb9467fc09030e5f44"
server_key_iv: "3e6c9cc2abc50486d93ad0ba0878f6c5"
server_key_version: "encryptionkey@0"
I20260924 12:51:49.075081 24439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 12:51:49.078742 24454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:49.080227 24439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 12:51:49.080402 24439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d"
format_stamp: "Formatted at 2026-09-24 12:51:49 on dist-test-slave-9tm1"
server_key: "0fa276a9884513eb9467fc09030e5f44"
server_key_iv: "3e6c9cc2abc50486d93ad0ba0878f6c5"
server_key_version: "encryptionkey@0"
I20260924 12:51:49.080586 24439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:49.106544 24439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:49.111194 24439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:49.111479 24439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:49.130281 24439 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:39647
I20260924 12:51:49.130306 24505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:39647 every 8 connection(s)
I20260924 12:51:49.131716 24439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:49.134521 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24439
I20260924 12:51:49.134711 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:49.135221 2373 external_mini_cluster.cc:1428] Setting key 25885c83a26f39c1be4dd6232924756e
I20260924 12:51:49.136108 24506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 12:51:49 dist-test-slave-9tm1 krb5kdc[24426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:49.149312 24506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d: Bootstrap starting.
I20260924 12:51:49.153823 24506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:49.155164 24506 log.cc:824] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:49.158965 24506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d: No bootstrap required, opened a new log
I20260924 12:51:49.159045 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:49.137032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36893 (local address 127.2.81.126:39647)
0924 12:51:49.138192 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:49.138203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:49.138244 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:51:49.139770 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:49.139793 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:49.140237 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:49.140633 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:49.142116 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:49.143483 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:49.144780 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:49.145554 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:49.145752 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:49.150429 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:49.150446 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:49.150463 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:49.150511 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:49.154608 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:49.155394 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:49.155402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:49.155409 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:49.155595 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:49.155980 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:49.155984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:49.155987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:49.156608 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:49.157037 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:49.157553 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 12:51:49.158025 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":728,"threads_started":1}
I20260924 12:51:49.163978 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:49.168337 24506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39647 } }
I20260924 12:51:49.168865 24506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:49.168995 24506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8bf31cdec4c57b88f9c4961a8bb3d, State: Initialized, Role: FOLLOWER
I20260924 12:51:49.169665 24506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39647 } }
I20260924 12:51:49.169919 24506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:49.170055 24506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:49.170212 24506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:49.173146 24506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39647 } }
I20260924 12:51:49.174059 24506 leader_election.cc:302] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af8bf31cdec4c57b88f9c4961a8bb3d; no voters:
I20260924 12:51:49.174863 24506 leader_election.cc:288] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:49.174995 24516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:49.177359 24506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:49.178227 24516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [term 1 LEADER]: Becoming Leader. State: Replica: 7af8bf31cdec4c57b88f9c4961a8bb3d, State: Running, Role: LEADER
I20260924 12:51:49.178925 24516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39647 } }
I20260924 12:51:49.186444 24517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39647 } } }
I20260924 12:51:49.187315 24517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:49.187012 24518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af8bf31cdec4c57b88f9c4961a8bb3d. Latest consensus state: current_term: 1 leader_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af8bf31cdec4c57b88f9c4961a8bb3d" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39647 } } }
I20260924 12:51:49.187793 24518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:49.187896 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:49.168443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49819 (local address 127.2.81.126:39647)
0924 12:51:49.168651 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:49.168657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:49.168923 (+ 266us) server_negotiation.cc:407] Connection header received
0924 12:51:49.169151 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:49.169156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:49.169235 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:49.169349 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:49.170968 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:49.172150 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:49.173915 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:49.174607 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:49.174789 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:49.178011 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:49.178018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:49.178024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:49.178073 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:49.182505 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:49.183667 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:49.183674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:49.183678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:49.183902 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:49.184745 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:49.184752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:49.184756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:49.185514 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:49.187093 (+ 1579us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:49.187265 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 12:51:49.187535 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 12:51:49.193140 24527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:49.197860 24527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:49.210757 24527 catalog_manager.cc:1347] Generated new cluster ID: 916b8ec8a38d44cc85c099e9c82d7554
I20260924 12:51:49.210908 24527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:49.220323 24527 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:49.222319 24527 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:49.238989 24527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af8bf31cdec4c57b88f9c4961a8bb3d: Generated new TSK 0
I20260924 12:51:49.240151 24527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:49.245961 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24439
I20260924 12:51:49.247213 24501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:49.339460 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24439
2026-09-24T12:51:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:49 dist-test-slave-9tm1 krb5kdc[24538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:49 dist-test-slave-9tm1 krb5kdc[24538](info): set up 2 sockets
Sep 24 12:51:49 dist-test-slave-9tm1 krb5kdc[24538](info): commencing operation
krb5kdc: starting...
W20260924 12:51:50.409873 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:50 dist-test-slave-9tm1 krb5kdc[24538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:50.677165 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:40049
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:51.014976 24551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:51.015308 24551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:51.015442 24551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:51.022902 24551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:51.023039 24551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:51.023099 24551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:51.023178 24551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:51.023222 24551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:51.033488 24551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:40049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:51.034785 24551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:51.036038 24551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:51.047636 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:51.047560 24557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:51.047602 24559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:51.048662 24551 server_base.cc:1034] running on GCE node
I20260924 12:51:51.049520 24551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:51.051060 24551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:51.052305 24551 hybrid_clock.cc:648] HybridClock initialized: now 1790254311052256 us; error 52 us; skew 500 ppm
I20260924 12:51:51.052700 24551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:51 dist-test-slave-9tm1 krb5kdc[24538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:51.055956 24551 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:51.057768 24551 webserver.cc:460] Webserver started at http://127.2.81.126:42979/ using document root <none> and password file <none>
I20260924 12:51:51.058465 24551 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:51.058595 24551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:51.058892 24551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:51.062057 24551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "75cf590063e249ceba3064cd5891ad93"
format_stamp: "Formatted at 2026-09-24 12:51:51 on dist-test-slave-9tm1"
server_key: "438afa861f957780eb2296328f53c2fe"
server_key_iv: "3bbbc8e815b1df523a9617ae4d95574d"
server_key_version: "encryptionkey@0"
I20260924 12:51:51.062849 24551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "75cf590063e249ceba3064cd5891ad93"
format_stamp: "Formatted at 2026-09-24 12:51:51 on dist-test-slave-9tm1"
server_key: "438afa861f957780eb2296328f53c2fe"
server_key_iv: "3bbbc8e815b1df523a9617ae4d95574d"
server_key_version: "encryptionkey@0"
I20260924 12:51:51.068320 24551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 12:51:51.072173 24566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:51.073764 24551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:51:51.073940 24551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "75cf590063e249ceba3064cd5891ad93"
format_stamp: "Formatted at 2026-09-24 12:51:51 on dist-test-slave-9tm1"
server_key: "438afa861f957780eb2296328f53c2fe"
server_key_iv: "3bbbc8e815b1df523a9617ae4d95574d"
server_key_version: "encryptionkey@0"
I20260924 12:51:51.074185 24551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:51.088615 24551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:51.093103 24551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:51.093384 24551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:51.111565 24551 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:40049
I20260924 12:51:51.111639 24617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:40049 every 8 connection(s)
I20260924 12:51:51.113039 24551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:51.116851 24618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:51.122611 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24551
I20260924 12:51:51.122910 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:51.123548 2373 external_mini_cluster.cc:1428] Setting key 69a0d0ac35bf5daac108bc18a579e8d4
I20260924 12:51:51.128539 24618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93: Bootstrap starting.
I20260924 12:51:51.132197 24618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:51.133445 24618 log.cc:824] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:51.136626 24618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93: No bootstrap required, opened a new log
I20260924 12:51:51.137246 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:51.125605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34981 (local address 127.2.81.126:40049)
0924 12:51:51.126468 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:51.126480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:51.126526 (+ 46us) server_negotiation.cc:407] Connection header received
0924 12:51:51.128238 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:51.128262 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:51.128710 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:51.129104 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:51.129636 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:51.130981 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:51.132752 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:51.133513 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:51.133703 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:51.134953 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:51.134961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:51.134976 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:51.135525 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:51.135602 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:51.135629 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:51.135726 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:51.135945 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 12:51:51.136194 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":375,"threads_started":1}
I20260924 12:51:51.140758 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:51.146034 24618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cf590063e249ceba3064cd5891ad93" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 40049 } }
I20260924 12:51:51.146559 24618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:51.146678 24618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cf590063e249ceba3064cd5891ad93, State: Initialized, Role: FOLLOWER
I20260924 12:51:51.147307 24618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cf590063e249ceba3064cd5891ad93" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 40049 } }
I20260924 12:51:51.147549 24618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:51.147688 24618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:51.147868 24618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:51.150200 24618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cf590063e249ceba3064cd5891ad93" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 40049 } }
I20260924 12:51:51.150755 24618 leader_election.cc:302] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75cf590063e249ceba3064cd5891ad93; no voters:
I20260924 12:51:51.151422 24618 leader_election.cc:288] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:51.151582 24628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:51.152477 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:51.145904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58551 (local address 127.2.81.126:40049)
0924 12:51:51.146115 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:51.146121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:51.146686 (+ 565us) server_negotiation.cc:407] Connection header received
0924 12:51:51.146893 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:51.146896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:51.146963 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:51.147057 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:51.147625 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:51.148454 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:51.149823 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:51.150280 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:51.150398 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:51.151535 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:51.151542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:51.151547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:51.151575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:51.151626 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:51.151649 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:51.151716 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:51.151964 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 12:51:51.152090 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 12:51:51.153244 24628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [term 1 LEADER]: Becoming Leader. State: Replica: 75cf590063e249ceba3064cd5891ad93, State: Running, Role: LEADER
I20260924 12:51:51.153947 24628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cf590063e249ceba3064cd5891ad93" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 40049 } }
I20260924 12:51:51.154281 24618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:51.156903 24629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75cf590063e249ceba3064cd5891ad93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cf590063e249ceba3064cd5891ad93" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 40049 } } }
I20260924 12:51:51.157356 24629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:51.157868 24630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75cf590063e249ceba3064cd5891ad93. Latest consensus state: current_term: 1 leader_uuid: "75cf590063e249ceba3064cd5891ad93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75cf590063e249ceba3064cd5891ad93" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 40049 } } }
I20260924 12:51:51.158116 24630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:51.166965 24636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:51.172755 24636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:51.183277 24636 catalog_manager.cc:1347] Generated new cluster ID: ca346ed56c6740049fd0958815ea8c67
I20260924 12:51:51.183465 24636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:51.199107 24636 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:51.200892 24636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:51.209280 24636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75cf590063e249ceba3064cd5891ad93: Generated new TSK 0
I20260924 12:51:51.210220 24636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:51.216472 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24551
W20260924 12:51:51.217635 24613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34981
W20260924 12:51:51.218283 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:51.218436 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24551
2026-09-24T12:51:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:51 dist-test-slave-9tm1 krb5kdc[24649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:51 dist-test-slave-9tm1 krb5kdc[24649](info): set up 2 sockets
Sep 24 12:51:51 dist-test-slave-9tm1 krb5kdc[24649](info): commencing operation
krb5kdc: starting...
W20260924 12:51:52.257467 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:52 dist-test-slave-9tm1 krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:52.513317 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36889
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:35343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:36889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:52.843616 24662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:52.843945 24662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:52.844055 24662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:52.851665 24662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:52.851804 24662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:52.851850 24662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:52.851908 24662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:52.851943 24662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:52.862212 24662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:35343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:36889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:52.863476 24662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:52.864741 24662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:52.876343 24667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:52.876381 24668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:52.876721 24670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:52.877293 24662 server_base.cc:1034] running on GCE node
I20260924 12:51:52.878157 24662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:52.879621 24662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:52.880872 24662 hybrid_clock.cc:648] HybridClock initialized: now 1790254312880820 us; error 45 us; skew 500 ppm
I20260924 12:51:52.881273 24662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:52 dist-test-slave-9tm1 krb5kdc[24649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:52.884514 24662 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:52.886363 24662 webserver.cc:460] Webserver started at http://127.2.81.126:37703/ using document root <none> and password file <none>
I20260924 12:51:52.887238 24662 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:52.887357 24662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:52.887620 24662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:52.891316 24662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "127c65aa4d2d488fabe47538a42c14f7"
format_stamp: "Formatted at 2026-09-24 12:51:52 on dist-test-slave-9tm1"
server_key: "55518c8ee56a0d7de2e4662e521f2650"
server_key_iv: "2d9a1556a1ee0f658c19ae4849a1a9f0"
server_key_version: "encryptionkey@0"
I20260924 12:51:52.892211 24662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "127c65aa4d2d488fabe47538a42c14f7"
format_stamp: "Formatted at 2026-09-24 12:51:52 on dist-test-slave-9tm1"
server_key: "55518c8ee56a0d7de2e4662e521f2650"
server_key_iv: "2d9a1556a1ee0f658c19ae4849a1a9f0"
server_key_version: "encryptionkey@0"
I20260924 12:51:52.897677 24662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:51:52.901355 24677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:52.902738 24662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:51:52.902906 24662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "127c65aa4d2d488fabe47538a42c14f7"
format_stamp: "Formatted at 2026-09-24 12:51:52 on dist-test-slave-9tm1"
server_key: "55518c8ee56a0d7de2e4662e521f2650"
server_key_iv: "2d9a1556a1ee0f658c19ae4849a1a9f0"
server_key_version: "encryptionkey@0"
I20260924 12:51:52.903095 24662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:52.909767 24662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:52.914448 24662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:52.914770 24662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:52.932510 24662 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:36889
I20260924 12:51:52.932569 24728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:36889 every 8 connection(s)
I20260924 12:51:52.933980 24662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:52.936687 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24662
I20260924 12:51:52.936867 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:52.937387 2373 external_mini_cluster.cc:1428] Setting key 7f7ba6a4cf402757c8ce4c0478350c7a
I20260924 12:51:52.938035 24729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:52.945566 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:52.939917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41257 (local address 127.2.81.126:36889)
0924 12:51:52.940656 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:52.940667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:52.940706 (+ 39us) server_negotiation.cc:407] Connection header received
0924 12:51:52.942224 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:52.942245 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:52.942585 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:52.942945 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:52.943902 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:52.943909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:52.943924 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:52.943941 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:52.944026 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:52.944055 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:52.944257 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:52.944390 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 12:51:52.944579 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":318,"threads_started":1}
I20260924 12:51:52.949116 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:52.951396 24729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7: Bootstrap starting.
I20260924 12:51:52.955147 24729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:52.955685 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:52.953192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50271 (local address 127.2.81.126:36889)
0924 12:51:52.953379 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:52.953384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:52.953732 (+ 348us) server_negotiation.cc:407] Connection header received
0924 12:51:52.953896 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:52.953900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:52.953951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:52.954048 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:52.955000 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:52.955006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:52.955010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 12:51:52.955027 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:52.955073 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 12:51:52.955090 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:52.955209 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:52.955343 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 12:51:52.955397 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:51:52.956353 24729 log.cc:824] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:52.959086 24729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7: No bootstrap required, opened a new log
I20260924 12:51:52.966300 24729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127c65aa4d2d488fabe47538a42c14f7" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36889 } }
I20260924 12:51:52.966727 24729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 12:51:52.966820 24729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127c65aa4d2d488fabe47538a42c14f7, State: Initialized, Role: FOLLOWER
I20260924 12:51:52.967388 24729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127c65aa4d2d488fabe47538a42c14f7" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36889 } }
I20260924 12:51:52.967618 24729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 12:51:52.967754 24729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 12:51:52.967916 24729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:52.970243 24729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127c65aa4d2d488fabe47538a42c14f7" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36889 } }
I20260924 12:51:52.970785 24729 leader_election.cc:302] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127c65aa4d2d488fabe47538a42c14f7; no voters:
I20260924 12:51:52.971405 24729 leader_election.cc:288] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 12:51:52.971534 24739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:52.972483 24739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [term 1 LEADER]: Becoming Leader. State: Replica: 127c65aa4d2d488fabe47538a42c14f7, State: Running, Role: LEADER
I20260924 12:51:52.973047 24739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127c65aa4d2d488fabe47538a42c14f7" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36889 } }
I20260924 12:51:52.973572 24729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:52.976006 24740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127c65aa4d2d488fabe47538a42c14f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127c65aa4d2d488fabe47538a42c14f7" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36889 } } }
I20260924 12:51:52.976145 24741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127c65aa4d2d488fabe47538a42c14f7. Latest consensus state: current_term: 1 leader_uuid: "127c65aa4d2d488fabe47538a42c14f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127c65aa4d2d488fabe47538a42c14f7" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 36889 } } }
I20260924 12:51:52.976542 24740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:52.976603 24741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7 [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:52.979095 24745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:52.985478 24745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:52.995769 24745 catalog_manager.cc:1347] Generated new cluster ID: e355d53cf66640a089aa4105916c826b
I20260924 12:51:52.995908 24745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:53.018461 24745 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:53.018935 24745 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:53.028597 24745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127c65aa4d2d488fabe47538a42c14f7: Generated new TSK 0
I20260924 12:51:53.029820 24745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:53.035930 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24662
W20260924 12:51:53.037659 24724 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41257
W20260924 12:51:53.038462 2373 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 12:51:53.038605 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24662
2026-09-24T12:51:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1820 ms)
[----------] 18 tests from AuthTokenIssuingTest (34493 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:53 dist-test-slave-9tm1 krb5kdc[24760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:53 dist-test-slave-9tm1 krb5kdc[24760](info): set up 2 sockets
Sep 24 12:51:53 dist-test-slave-9tm1 krb5kdc[24760](info): commencing operation
krb5kdc: starting...
W20260924 12:51:54.077639 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:54 dist-test-slave-9tm1 krb5kdc[24760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:54.345338 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:43331
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:43331,127.2.81.125:40459,127.2.81.124:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:54.695986 24773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:51:54.696375 24773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:54.696475 24773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:54.696589 24773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:54.704243 24773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:54.704361 24773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:54.704444 24773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:54.704509 24773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:54.704598 24773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:54.714957 24773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:43331,127.2.81.125:40459,127.2.81.124:34165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:54.716260 24773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:54.717533 24773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:54.729430 24778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:54.729405 24779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:54.729595 24781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:54.730648 24773 server_base.cc:1034] running on GCE node
I20260924 12:51:54.731501 24773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:54.733078 24773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:54.734318 24773 hybrid_clock.cc:648] HybridClock initialized: now 1790254314734241 us; error 54 us; skew 500 ppm
I20260924 12:51:54.734719 24773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:54 dist-test-slave-9tm1 krb5kdc[24760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:54.747483 24773 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:54.749258 24773 webserver.cc:460] Webserver started at http://127.2.81.126:36397/ using document root <none> and password file <none>
I20260924 12:51:54.750017 24773 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:54.750146 24773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:54.750464 24773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:54.753796 24773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "5bd5c62df2d242088d9783f845d1f39b"
format_stamp: "Formatted at 2026-09-24 12:51:54 on dist-test-slave-9tm1"
server_key: "94e42840812bc8caa47e58e281bb4fc2"
server_key_iv: "3348fd7b9d35489a8d4df0d2357ccf43"
server_key_version: "encryptionkey@0"
I20260924 12:51:54.754645 24773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "5bd5c62df2d242088d9783f845d1f39b"
format_stamp: "Formatted at 2026-09-24 12:51:54 on dist-test-slave-9tm1"
server_key: "94e42840812bc8caa47e58e281bb4fc2"
server_key_iv: "3348fd7b9d35489a8d4df0d2357ccf43"
server_key_version: "encryptionkey@0"
I20260924 12:51:54.760082 24773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 12:51:54.763878 24788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:54.765456 24773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 12:51:54.765659 24773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "5bd5c62df2d242088d9783f845d1f39b"
format_stamp: "Formatted at 2026-09-24 12:51:54 on dist-test-slave-9tm1"
server_key: "94e42840812bc8caa47e58e281bb4fc2"
server_key_iv: "3348fd7b9d35489a8d4df0d2357ccf43"
server_key_version: "encryptionkey@0"
I20260924 12:51:54.765887 24773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:54.787945 24773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:54.792817 24773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:54.793124 24773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:54.811960 24773 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:43331
I20260924 12:51:54.811973 24839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:43331 every 8 connection(s)
I20260924 12:51:54.813385 24773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:54.817090 24840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:54.823478 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24773
I20260924 12:51:54.823680 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:54.824267 2373 external_mini_cluster.cc:1428] Setting key bece026aab01e2e08e5472c8ab9165e8
I20260924 12:51:54.823730 24840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:54 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.125@KRBTEST.COM; defaulting to no policy
I20260924 12:51:54.846254 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.826926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:43331 (local address 127.0.0.1:42056)
0924 12:51:54.827786 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.827829 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:54.829870 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:54.830715 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:54.830722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:54.831403 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:54.831738 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:54.831755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:54.833368 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:54.833371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:54.834689 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:54.834703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:54.835978 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:54.835988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:54.836164 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:54.836779 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:54.836804 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:54.839005 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:54.842149 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:54.842160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:54.842173 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:54.842561 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:54.842956 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:54.842961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:54.842965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:54.843113 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:54.843880 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:54.843888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:54.844081 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 12:51:54.844305 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 12:51:54.844915 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":144,"thread_start_us":377,"threads_started":1}
I20260924 12:51:54.846287 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.826926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.2.81.126:43331)
0924 12:51:54.828558 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:54.828566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:54.828619 (+ 53us) server_negotiation.cc:407] Connection header received
0924 12:51:54.830244 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:54.830250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:54.830495 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:54.830677 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:54.831928 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:54.833228 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:54.834967 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:54.835570 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:54.835761 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:54.839238 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:54.839247 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:54.839260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:54.839293 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:54.842023 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:54.842684 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:54.842692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:54.842695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:54.842852 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:54.843240 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:54.843243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:54.843245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:54.843748 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:54.843939 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:54.844475 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 12:51:54.844754 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":1225,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.125@KRBTEST.COM" created.
I20260924 12:51:54.849363 24840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:51:54.850386 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.850015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40324)
0924 12:51:54.850166 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.850271 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 12:51:54.850601 24790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
I20260924 12:51:54.852036 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.851692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40330)
0924 12:51:54.851840 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.851904 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 12:51:54.852226 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:40459: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab.
Entry for principal kudu/127.2.81.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab.
I20260924 12:51:54.879908 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:40459
--webserver_interface=127.2.81.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab
--principal=kudu/127.2.81.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:43331,127.2.81.125:40459,127.2.81.124:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:51:54.905076 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } attempt: 1
I20260924 12:51:54.906381 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.905857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40338)
0924 12:51:54.906064 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.906202 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 12:51:54.907585 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.907143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40348)
0924 12:51:54.907318 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.907402 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 12:51:54.907786 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:40459: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
I20260924 12:51:54.976109 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } attempt: 2
I20260924 12:51:54.977384 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.976856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40350)
0924 12:51:54.977143 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.977233 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 12:51:54.978437 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:54.978066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40366)
0924 12:51:54.978213 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:54.978296 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 12:51:54.978628 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:40459: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
I20260924 12:51:55.069897 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } attempt: 3
I20260924 12:51:55.071064 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.070619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40370)
0924 12:51:55.070821 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.070912 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 12:51:55.072075 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.071708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40384)
0924 12:51:55.071852 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.071923 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 12:51:55.072245 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:40459: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
W20260924 12:51:55.215415 24847 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260924 12:51:55.215531 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } attempt: 4
W20260924 12:51:55.215772 24847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:55.215875 24847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:55.215983 24847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 12:51:55.216693 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.216184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40400)
0924 12:51:55.216468 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.216557 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 12:51:55.217682 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.217323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40402)
0924 12:51:55.217466 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.217551 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 12:51:55.217862 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:40459: Network error: Client connection negotiation failed: client connection to 127.2.81.125:40459: connect: Connection refused (error 111)
W20260924 12:51:55.223750 24847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:55.223855 24847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:55.223942 24847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:55.224006 24847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:55.224087 24847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:55.234424 24847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:43331,127.2.81.125:40459,127.2.81.124:34165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab
--principal=kudu/127.2.81.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.81.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:55.235791 24847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:55.237166 24847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:55.249564 24852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:55.249583 24855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:55.249475 24853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:55.250111 24847 server_base.cc:1034] running on GCE node
I20260924 12:51:55.250886 24847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:55.252321 24847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:55.253572 24847 hybrid_clock.cc:648] HybridClock initialized: now 1790254315253512 us; error 35 us; skew 500 ppm
I20260924 12:51:55.253943 24847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:55.257189 24847 init.cc:377] Logged in from keytab as kudu/127.2.81.125@KRBTEST.COM (short username kudu)
I20260924 12:51:55.258718 24847 webserver.cc:460] Webserver started at http://127.2.81.125:44207/ using document root <none> and password file <none>
I20260924 12:51:55.259402 24847 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:55.259491 24847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:55.259759 24847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:55.263049 24847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data/instance:
uuid: "568b482e70c249c698a973680e38ce60"
format_stamp: "Formatted at 2026-09-24 12:51:55 on dist-test-slave-9tm1"
server_key: "71e36ed1d316e3faccdd53a239e2e0fc"
server_key_iv: "b9f9c9070d4605ef48eecf06b1b6ba18"
server_key_version: "encryptionkey@0"
I20260924 12:51:55.263908 24847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal/instance:
uuid: "568b482e70c249c698a973680e38ce60"
format_stamp: "Formatted at 2026-09-24 12:51:55 on dist-test-slave-9tm1"
server_key: "71e36ed1d316e3faccdd53a239e2e0fc"
server_key_iv: "b9f9c9070d4605ef48eecf06b1b6ba18"
server_key_version: "encryptionkey@0"
I20260924 12:51:55.269600 24847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 12:51:55.273373 24862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:55.274801 24847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 12:51:55.274967 24847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal
uuid: "568b482e70c249c698a973680e38ce60"
format_stamp: "Formatted at 2026-09-24 12:51:55 on dist-test-slave-9tm1"
server_key: "71e36ed1d316e3faccdd53a239e2e0fc"
server_key_iv: "b9f9c9070d4605ef48eecf06b1b6ba18"
server_key_version: "encryptionkey@0"
I20260924 12:51:55.275183 24847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:55.295104 24847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:55.299738 24847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:55.300050 24847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:55.317610 24847 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.125:40459
I20260924 12:51:55.317634 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.125:40459 every 8 connection(s)
I20260924 12:51:55.319003 24847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
I20260924 12:51:55.323387 24914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:55.324934 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24847
I20260924 12:51:55.325244 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-1/wal/instance
I20260924 12:51:55.325798 2373 external_mini_cluster.cc:1428] Setting key 5bc944fbf93cc9d0e6f7798813c8cad6
I20260924 12:51:55.331960 24914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.124@KRBTEST.COM" created.
I20260924 12:51:55.354514 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.335670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.2.81.126:43331)
0924 12:51:55.335925 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.335933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.337524 (+ 1591us) server_negotiation.cc:407] Connection header received
0924 12:51:55.339609 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.339616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.339710 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.339827 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.341607 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.342603 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.344231 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.344646 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.344804 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.348393 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.348398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.348402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.348439 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.351248 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.352174 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.352179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.352181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.352306 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.352759 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.352762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.352764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.353102 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.353220 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.354101 (+ 881us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.354283 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 12:51:55.355779 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.336299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:43331 (local address 127.0.0.1:42066)
0924 12:51:55.337285 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.337365 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.339285 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.340047 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.340057 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.340680 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.341391 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.341414 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.342774 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.342778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.344035 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.344050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.344925 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.344932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.345214 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.345872 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.345896 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.348123 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.351414 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.351424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.351444 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.352034 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.352401 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.352406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.352409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.352648 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.353217 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.353225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.353719 (+ 494us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.353996 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.354484 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":389,"threads_started":1}
I20260924 12:51:55.357688 24914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:51:55.372803 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.358474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40408)
0924 12:51:55.358674 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.358696 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.358786 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.359792 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.359796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.360238 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.360369 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.360378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.361801 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.361806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.362767 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.362781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.363617 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.363623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.363842 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.364394 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.364412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.366607 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.369875 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.369881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.369885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.370397 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.370820 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.370825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.370829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.371014 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.371690 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.371695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.371860 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.372022 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.372330 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":37}
I20260924 12:51:55.372836 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.358741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.2.81.125:40459)
0924 12:51:55.359366 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.359373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.359399 (+ 26us) server_negotiation.cc:407] Connection header received
0924 12:51:55.359442 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.359448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.359625 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.359777 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.360533 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.361672 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.362926 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.363477 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.363593 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.366832 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.366848 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.366865 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.366901 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.369726 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.370546 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.370556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.370560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.370705 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.371160 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.371165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.371167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.371540 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.371748 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.372254 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.372459 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":484,"thread_start_us":351,"threads_started":1}
I20260924 12:51:55.375548 24914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:51:55.376716 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.376233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60382)
0924 12:51:55.376377 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.376499 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 12:51:55.376994 24864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.378647 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.378217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60388)
0924 12:51:55.378398 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.378503 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 12:51:55.378883 24914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab.
Entry for principal kudu/127.2.81.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab.
I20260924 12:51:55.384831 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:34165
--webserver_interface=127.2.81.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab
--principal=kudu/127.2.81.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:36083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:43331,127.2.81.125:40459,127.2.81.124:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:51:55.428184 24914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 1
I20260924 12:51:55.429419 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.428920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60400)
0924 12:51:55.429153 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.429268 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 12:51:55.430562 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.430143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60414)
0924 12:51:55.430321 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.430402 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 12:51:55.430781 24914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.499076 24914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 2
I20260924 12:51:55.500330 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.499843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60430)
0924 12:51:55.500070 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.500166 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 12:51:55.501586 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.501189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60440)
0924 12:51:55.501340 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.501418 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 12:51:55.501803 24914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.517105 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } attempt: 5
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:51:55.531013 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.517789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40412)
0924 12:51:55.518019 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.518051 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.518183 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.518549 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.518555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.519066 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.519311 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.519325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.520291 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.520297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.521685 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.521701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.522429 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.522435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.522667 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.523306 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.523329 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.525751 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.528599 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.528605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.528609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.529028 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.529381 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.529387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.529391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.529532 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.530049 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.530054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.530217 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.530353 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.530647 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 12:51:55.531013 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.517984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.2.81.125:40459)
0924 12:51:55.518141 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.518145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.518164 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:51:55.518328 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.518332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.518397 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.518483 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.519473 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.520153 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.521830 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.522271 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.522365 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.525962 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.525967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.525971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.526005 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.528452 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.529140 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.529146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.529148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.529272 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.529642 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.529646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.529648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.529935 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.530065 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.530495 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.530708 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 12:51:55.532001 24840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:51:55.532922 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.532547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60456)
0924 12:51:55.532702 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.532779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 12:51:55.533788 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.533471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60466)
0924 12:51:55.533613 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.533693 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 12:51:55.533967 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.585193 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 1
I20260924 12:51:55.586516 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.586012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60480)
0924 12:51:55.586253 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.586341 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 12:51:55.587545 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.587227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60492)
0924 12:51:55.587361 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.587434 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 12:51:55.587783 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.593102 24914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 3
I20260924 12:51:55.594169 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.593722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60508)
0924 12:51:55.593911 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.594008 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 12:51:55.595788 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.595060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60514)
0924 12:51:55.595543 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.595644 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
W20260924 12:51:55.596029 24914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.656009 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 2
I20260924 12:51:55.657214 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.656706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60520)
0924 12:51:55.656907 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.657052 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 12:51:55.658327 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.657973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60532)
0924 12:51:55.658131 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.658208 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 12:51:55.658528 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
W20260924 12:51:55.730439 24921 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:51:55.730789 24921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:55.730860 24921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:55.730943 24921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:55.739219 24921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:55.739336 24921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:55.739393 24921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:55.739480 24921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:55.739564 24921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:55.739436 24914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 4
I20260924 12:51:55.740731 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.740172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60536)
0924 12:51:55.740426 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.740529 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 12:51:55.741771 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.741407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60546)
0924 12:51:55.741574 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.741652 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 12:51:55.742003 24914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.751487 24921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:36083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:43331,127.2.81.125:40459,127.2.81.124:34165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab
--principal=kudu/127.2.81.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.81.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:55.752735 24921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:55.754082 24921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 12:51:55.764719 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 3
W20260924 12:51:55.765774 24927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:55.765856 24926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:55.766036 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.765516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60562)
0924 12:51:55.765738 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.765835 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 12:51:55.766083 24929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:55.766573 24921 server_base.cc:1034] running on GCE node
I20260924 12:51:55.767239 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.766833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60564)
0924 12:51:55.767021 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.767099 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 12:51:55.767472 24840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:34165: Network error: Client connection negotiation failed: client connection to 127.2.81.124:34165: connect: Connection refused (error 111)
I20260924 12:51:55.767540 24921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:55.769006 24921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:55.770242 24921 hybrid_clock.cc:648] HybridClock initialized: now 1790254315770166 us; error 56 us; skew 500 ppm
I20260924 12:51:55.770566 24921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:55.774433 24921 init.cc:377] Logged in from keytab as kudu/127.2.81.124@KRBTEST.COM (short username kudu)
I20260924 12:51:55.776331 24921 webserver.cc:460] Webserver started at http://127.2.81.124:37525/ using document root <none> and password file <none>
I20260924 12:51:55.777596 24921 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:55.777760 24921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:55.778205 24921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:55.782121 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data/instance:
uuid: "dae0a0457da24016955ec1b19b954717"
format_stamp: "Formatted at 2026-09-24 12:51:55 on dist-test-slave-9tm1"
server_key: "40934cf8adf7e719b994807e6ad6bbc9"
server_key_iv: "88eb4144e5668a4f56e0e74657c98e27"
server_key_version: "encryptionkey@0"
I20260924 12:51:55.783288 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal/instance:
uuid: "dae0a0457da24016955ec1b19b954717"
format_stamp: "Formatted at 2026-09-24 12:51:55 on dist-test-slave-9tm1"
server_key: "40934cf8adf7e719b994807e6ad6bbc9"
server_key_iv: "88eb4144e5668a4f56e0e74657c98e27"
server_key_version: "encryptionkey@0"
I20260924 12:51:55.790110 24921 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260924 12:51:55.794252 24936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:55.796271 24921 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 12:51:55.796486 24921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal
uuid: "dae0a0457da24016955ec1b19b954717"
format_stamp: "Formatted at 2026-09-24 12:51:55 on dist-test-slave-9tm1"
server_key: "40934cf8adf7e719b994807e6ad6bbc9"
server_key_iv: "88eb4144e5668a4f56e0e74657c98e27"
server_key_version: "encryptionkey@0"
I20260924 12:51:55.796739 24921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:55.819658 24921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:55.824020 24921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:55.824290 24921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:55.842690 24921 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.124:34165
I20260924 12:51:55.842713 24987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.124:34165 every 8 connection(s)
I20260924 12:51:55.844319 24921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
I20260924 12:51:55.848186 24988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:55.850816 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 24921
I20260924 12:51:55.850993 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790253949676453-2373-0/minicluster-data/master-2/wal/instance
I20260924 12:51:55.851488 2373 external_mini_cluster.cc:1428] Setting key 6ab966d287ddcd3393beaa5440fc91e3
I20260924 12:51:55.856479 24988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:55.875278 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.858936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.2.81.126:43331)
0924 12:51:55.859429 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.859435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.859992 (+ 557us) server_negotiation.cc:407] Connection header received
0924 12:51:55.860163 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.860169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.860249 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.860364 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.861776 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.862882 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.864777 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.865232 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.865329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.868885 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.868891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.868894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.868930 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.872041 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.872949 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.872954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.872957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.873133 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.873563 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.873567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.873570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.873885 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.874041 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.874648 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.874871 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
I20260924 12:51:55.875766 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.858944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:43331 (local address 127.0.0.1:42068)
0924 12:51:55.859842 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.859890 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.860047 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.860437 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.860444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.861330 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.861631 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.861643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.863058 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.863062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.864618 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.864633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.865465 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.865473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.865656 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.866277 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.866305 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.868670 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.872194 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.872204 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.872210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.872823 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.873282 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.873286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.873288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.873437 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.874047 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.874055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.874256 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.874464 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.874801 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":499,"threads_started":1}
I20260924 12:51:55.875788 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.853754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.2.81.124:34165)
0924 12:51:55.854597 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.854610 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.854651 (+ 41us) server_negotiation.cc:407] Connection header received
0924 12:51:55.856339 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.856363 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.856787 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.857208 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.858634 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.860286 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.862027 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.863049 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.863227 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.867747 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.867755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.867770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.867800 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.871343 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.872089 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.872098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.872108 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.872290 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.872754 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.872757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.872759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.873463 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.873777 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.874179 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.874563 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":402,"threads_started":1}
I20260924 12:51:55.877372 24988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:55.892256 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.878125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.2.81.125:40459)
0924 12:51:55.878321 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.878326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.878851 (+ 525us) server_negotiation.cc:407] Connection header received
0924 12:51:55.879046 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.879052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.879122 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.879250 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.880180 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.880962 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.882306 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.882899 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.883005 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.886169 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.886176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.886179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.886214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.889082 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.889917 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.889923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.889927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.890107 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.890615 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.890622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.890626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.891047 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.891282 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.891727 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.891920 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:51:55.892278 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.878187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:40459 (local address 127.0.0.1:40418)
0924 12:51:55.878747 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.878763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.878924 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.879288 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.879293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.879787 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.879988 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.880000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.881129 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.881133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.882130 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.882145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.883058 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.883064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.883276 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.883833 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.883852 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.885964 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.889230 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.889234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.889237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.889729 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.890271 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.890275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.890278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.890427 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.891222 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.891228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.891418 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.891595 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.891929 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471}
I20260924 12:51:55.893492 24988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:51:55.894375 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.879093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.2.81.126:43331)
0924 12:51:55.879267 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.879272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.879312 (+ 40us) server_negotiation.cc:407] Connection header received
0924 12:51:55.879491 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.879495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.879575 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.879667 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.880909 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.881847 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.883205 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.883626 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.883751 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.887708 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.887714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.887718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.887761 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.891286 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.892227 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.892234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.892238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.892400 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.892821 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.892828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.892830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.893218 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.893502 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.893821 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.894063 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:51:55.907481 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.894190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60580)
0924 12:51:55.894355 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.894373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.894470 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.894757 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.894760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.895178 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.895326 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.895338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.896551 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.896556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.897507 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.897519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.898309 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.898315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.898538 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.899294 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.899316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.901460 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.905097 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.905101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.905104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.905445 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.905823 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.905827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.905830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.905992 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.906603 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.906606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.906742 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.906850 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.907098 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 12:51:55.907584 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.894350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.2.81.124:34165)
0924 12:51:55.894492 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.894497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.894514 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:51:55.894554 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.894558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.894620 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.894708 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.895483 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.896431 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.897661 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.898155 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.898260 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.901699 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.901705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.901710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.901745 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.904926 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.905585 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.905589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.905592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.905713 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.906123 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.906128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.906130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.906483 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.906638 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.906959 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.907273 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":50}
I20260924 12:51:55.912207 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: Bootstrap starting.
I20260924 12:51:55.912639 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.896176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.2.81.125:40459)
0924 12:51:55.896359 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.896364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.896378 (+ 14us) server_negotiation.cc:407] Connection header received
0924 12:51:55.896420 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.896423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.896478 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.896568 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.897857 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.898880 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.900282 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.900793 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.900906 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.906588 (+ 5682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.906597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.906602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.906648 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.909969 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.910697 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.910702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.910704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.910843 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.911231 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.911235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.911237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.911525 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.911777 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.912078 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.912320 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:51:55.915851 24988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:55.917116 24988 log.cc:824] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:55.917722 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:51:55.920258 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: No bootstrap required, opened a new log
I20260924 12:51:55.921154 24815 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5bd5c62df2d242088d9783f845d1f39b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42078
I20260924 12:51:55.928961 24988 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:55.929500 24988 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae0a0457da24016955ec1b19b954717, State: Initialized, Role: FOLLOWER
I20260924 12:51:55.929958 24988 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:55.930861 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:55.931162 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:55.932109 24988 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 12:51:55.941248 25006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:51:55.941381 25006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:51:55.944698 24840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 4
I20260924 12:51:55.939750 24921 master_runner.cc:186] Error getting master registration for 127.2.81.126:43331: OK, instance_id { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" instance_seqno: 1790254314806794 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 12:51:55.951326 24921 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 12:51:55 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
I20260924 12:51:55.961941 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.945474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60584)
0924 12:51:55.945755 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:55.945774 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:55.945908 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:55.946287 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:55.946291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:55.946699 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:55.946866 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.946876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.948424 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.948430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.949434 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:55.949449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.950564 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.950571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:55.950766 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.951405 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:55.951427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:55.954005 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:55.958474 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.958480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.958484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.959204 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.959769 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:55.959775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:55.959779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:55.959974 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:55.960777 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:55.960783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:55.961027 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 12:51:55.961188 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 12:51:55.961536 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260924 12:51:55.961944 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:55.945539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.2.81.124:34165)
0924 12:51:55.945760 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:55.945765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:55.945833 (+ 68us) server_negotiation.cc:407] Connection header received
0924 12:51:55.946061 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:55.946066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:55.946150 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:55.946251 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:55.947022 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.948257 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.949598 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:55.950377 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:55.950530 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:55.954297 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:55.954305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:55.954310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:55.954359 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:55.958253 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.959387 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.959394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.959398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.959571 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:55.960115 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:55.960121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:55.960125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:55.960621 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:55.960784 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:55.961330 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 12:51:55.961570 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 12:51:55.967024 24840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: Bootstrap starting.
I20260924 12:51:55.970402 24840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:55.971531 24840 log.cc:824] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:55.974427 24840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: No bootstrap required, opened a new log
I20260924 12:51:55.979839 24840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:55.980206 24840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd5c62df2d242088d9783f845d1f39b, State: Initialized, Role: FOLLOWER
I20260924 12:51:55.980651 24840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:55.981616 25009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:55.982023 25009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:55.982869 24840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:55.983043 24815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:51:55.983327 24815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:55.987192 24815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:55.989821 24815 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [CANDIDATE]: Term 1 election: Requested vote from peers 568b482e70c249c698a973680e38ce60 (127.2.81.125:40459), dae0a0457da24016955ec1b19b954717 (127.2.81.124:34165)
I20260924 12:51:55.990758 24963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bd5c62df2d242088d9783f845d1f39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dae0a0457da24016955ec1b19b954717"
I20260924 12:51:55.991302 24963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:55.995139 24963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd5c62df2d242088d9783f845d1f39b in term 1.
I20260924 12:51:55.995916 24790 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd5c62df2d242088d9783f845d1f39b, dae0a0457da24016955ec1b19b954717; no voters:
I20260924 12:51:55.996520 25009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:51:56.008015 25009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [term 1 LEADER]: Becoming Leader. State: Replica: 5bd5c62df2d242088d9783f845d1f39b, State: Running, Role: LEADER
I20260924 12:51:56.008790 25009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:56.027935 25022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd5c62df2d242088d9783f845d1f39b. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.028841 25022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:56.029927 24773 master_runner.cc:186] Error getting master registration for 127.2.81.125:40459: OK, instance_id { permanent_uuid: "568b482e70c249c698a973680e38ce60" instance_seqno: 1790254315312585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 12:51:56.041240 24914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } attempt: 5
I20260924 12:51:56.041505 24773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 12:51:56.046124 24794 tablet.cc:2369] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 12:51:56 dist-test-slave-9tm1 krb5kdc[24760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
I20260924 12:51:56.059962 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.027280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.2.81.126:43331)
0924 12:51:56.027837 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.027843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.039844 (+ 12001us) server_negotiation.cc:407] Connection header received
0924 12:51:56.041078 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.041084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.041165 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.041239 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.044140 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.045324 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.047440 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.048132 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.048244 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.051947 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:56.051955 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:56.051960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:56.052012 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:56.056300 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.057352 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.057358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.057362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.057554 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.058020 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.058025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.058029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.058414 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:56.058686 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.059245 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.059485 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
W20260924 12:51:56.065480 25020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:51:56.065618 25020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:51:56.065846 25029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:51:56.068254 25029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:51:56.068639 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.041858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:34165 (local address 127.0.0.1:60612)
0924 12:51:56.042053 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:56.042073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:56.042188 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:56.043809 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:56.043815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:56.044359 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:56.044618 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:56.044632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.049322 (+ 4690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.049329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:56.050895 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:56.050917 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.052811 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.052819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:56.053104 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.053746 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:56.053773 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:56.057590 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:56.064433 (+ 6843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:56.064450 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:56.064454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:56.065057 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:56.065784 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:56.065789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:56.065793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:56.065983 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:56.067388 (+ 1405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:56.067394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:56.067587 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 12:51:56.067883 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 12:51:56.068217 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 12:51:56.070755 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.042241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.2.81.124:34165)
0924 12:51:56.043168 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.043175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.043228 (+ 53us) server_negotiation.cc:407] Connection header received
0924 12:51:56.043324 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.043353 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.043503 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.045243 (+ 1740us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.045767 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.049076 (+ 3309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.051222 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.052550 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.057871 (+ 5321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.058404 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:56.058413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:56.058418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:56.058483 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:56.064205 (+ 5722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.065293 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.065300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.065303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.065576 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.066169 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.066175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.066178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.067169 (+ 991us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:56.069171 (+ 2002us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.069413 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.069867 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":371,"threads_started":1}
I20260924 12:51:56.073853 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.027874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.2.81.125:40459)
0924 12:51:56.028060 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.028066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.035354 (+ 7288us) server_negotiation.cc:407] Connection header received
0924 12:51:56.037040 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.037048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.037148 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.037224 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.048479 (+ 11255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.049544 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.060030 (+ 10486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.060673 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.060779 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.066045 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:56.066052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:56.066057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:56.066098 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:56.069995 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.070911 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.070917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.070920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.071107 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.071570 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.071576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.071578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.071949 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:56.072200 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.073183 (+ 983us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.073453 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 12:51:56.076471 24942 tablet.cc:2369] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:51:56.076905 24914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: Bootstrap starting.
I20260924 12:51:56.083679 24914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: Neither blocks nor log segments found. Creating new log.
I20260924 12:51:56.085667 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.036005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.2.81.124:34165)
0924 12:51:56.036197 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.036203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.045530 (+ 9327us) server_negotiation.cc:407] Connection header received
0924 12:51:56.045958 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.045964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.046042 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.046108 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.057130 (+ 11022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.058149 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.064240 (+ 6091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.064905 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.065054 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.078016 (+ 12962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:56.078024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:56.078041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:56.078085 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:56.082084 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.083065 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.083075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.083078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.083260 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:56.083732 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:56.083737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:56.083740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:56.084137 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:56.084390 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.084942 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.085253 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:51:56.086295 24914 log.cc:824] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: Log is configured to *not* fsync() on all Append() calls
I20260924 12:51:56.091996 24914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: No bootstrap required, opened a new log
I20260924 12:51:56.098069 24963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [term 1 FOLLOWER]: Refusing update from remote peer 5bd5c62df2d242088d9783f845d1f39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:56.101449 25022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:56.105773 24914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:56.106339 24914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568b482e70c249c698a973680e38ce60, State: Initialized, Role: FOLLOWER
I20260924 12:51:56.106948 24914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } }
I20260924 12:51:56.115769 25036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.118113 25036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:56.117962 24914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:51:56.118367 24888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:51:56.122490 24888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [term 1 FOLLOWER]: Refusing update from remote peer 5bd5c62df2d242088d9783f845d1f39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:51:56.123215 24889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bd5c62df2d242088d9783f845d1f39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "568b482e70c249c698a973680e38ce60"
I20260924 12:51:56.123749 25022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:51:56.127004 24889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd5c62df2d242088d9783f845d1f39b in term 1.
I20260924 12:51:56.127104 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd5c62df2d242088d9783f845d1f39b. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.127417 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:56.142055 25033 mvcc.cc:205] Tried to move back new op lower bound from 7332881678724386816 to 7332881678380748800. Current Snapshot: MvccSnapshot[applied={T|T < 7332881678724386816}]
I20260924 12:51:56.142484 25022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.142751 25022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:56.143013 25035 mvcc.cc:205] Tried to move back new op lower bound from 7332881678724386816 to 7332881678380748800. Current Snapshot: MvccSnapshot[applied={T|T < 7332881678724386816}]
I20260924 12:51:56.144842 25029 catalog_manager.cc:1347] Generated new cluster ID: d69623a129dd43b288621aad2e701af1
I20260924 12:51:56.144955 25029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:51:56.161342 25036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd5c62df2d242088d9783f845d1f39b. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.161672 25036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:56.161604 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.161855 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:56.167649 25022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.167973 25022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b [sys.catalog]: This master's current role is: LEADER
I20260924 12:51:56.174655 25040 mvcc.cc:205] Tried to move back new op lower bound from 7332881678724386816 to 7332881678380748800. Current Snapshot: MvccSnapshot[applied={T|T < 7332881678724386816}]
I20260924 12:51:56.177762 25029 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:51:56.180223 25029 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:51:56.183279 25036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bd5c62df2d242088d9783f845d1f39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd5c62df2d242088d9783f845d1f39b" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 43331 } } peers { permanent_uuid: "568b482e70c249c698a973680e38ce60" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 40459 } } peers { permanent_uuid: "dae0a0457da24016955ec1b19b954717" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 34165 } } }
I20260924 12:51:56.183583 25036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:51:56.212641 25029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd5c62df2d242088d9783f845d1f39b: Generated new TSK 0
I20260924 12:51:56.213991 25029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:51:56.224669 25053 catalog_manager.cc:1259] Loaded cluster ID: d69623a129dd43b288621aad2e701af1
I20260924 12:51:56.228237 25053 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: loading cluster ID for follower catalog manager: success
I20260924 12:51:56.232849 25053 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: acquiring CA information for follower catalog manager: success
I20260924 12:51:56.238951 25053 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 568b482e70c249c698a973680e38ce60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 12:51:56.352695 24918 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.335618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.2.81.125:40459)
0924 12:51:56.335834 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.335840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.349987 (+ 14147us) server_negotiation.cc:407] Connection header received
0924 12:51:56.351187 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.351193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.351281 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.351356 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.352457 (+ 1101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40452: BlockingRecv error: recv got EOF from 127.0.0.1:40452 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 12:51:56.359249 24992 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.333368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.2.81.126:43331)
0924 12:51:56.333639 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.333646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.349587 (+ 15941us) server_negotiation.cc:407] Connection header received
0924 12:51:56.357423 (+ 7836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.357430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.357530 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.357612 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.358950 (+ 1338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42108: BlockingRecv error: recv got EOF from 127.0.0.1:42108 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 12:51:56.366307 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.350532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.2.81.124:34165)
0924 12:51:56.350734 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.350740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.354213 (+ 3473us) server_negotiation.cc:407] Connection header received
0924 12:51:56.354385 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.354390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.354477 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.354596 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.366035 (+ 11439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60616: BlockingRecv error: recv got EOF from 127.0.0.1:60616 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 12:51:56.425504 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.405842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.2.81.125:40459)
0924 12:51:56.406036 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.406042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.417081 (+ 11039us) server_negotiation.cc:407] Connection header received
0924 12:51:56.417675 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.417682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.417901 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.418066 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.418804 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.419963 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.422215 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.422923 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.423039 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.423879 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.424589 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.424739 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.425036 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.425150 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 12:51:56.472745 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.459359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.2.81.124:34165)
0924 12:51:56.459558 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.459564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.470296 (+ 10732us) server_negotiation.cc:407] Connection header received
0924 12:51:56.471067 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.471074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.471214 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.471296 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.472478 (+ 1182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60620: BlockingRecv error: recv got EOF from 127.0.0.1:60620 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 12:51:56.528008 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.509498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.2.81.125:40459)
0924 12:51:56.509812 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.509819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.520475 (+ 10656us) server_negotiation.cc:407] Connection header received
0924 12:51:56.521073 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.521080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.521183 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.521255 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.521969 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.523148 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.525228 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.525936 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.526042 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.527115 (+ 1073us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.527287 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.527350 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.527596 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.527693 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
W20260924 12:51:56.578235 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.564824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.2.81.124:34165)
0924 12:51:56.565034 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.565040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.575905 (+ 10865us) server_negotiation.cc:407] Connection header received
0924 12:51:56.576425 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.576432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.576535 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.576621 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.577738 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60628: BlockingRecv error: recv got EOF from 127.0.0.1:60628 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 12:51:56.630295 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.614755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.2.81.125:40459)
0924 12:51:56.614934 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.614940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.625545 (+ 10605us) server_negotiation.cc:407] Connection header received
0924 12:51:56.625750 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.625756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.625849 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.625943 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.626370 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.627146 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.628544 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.629045 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.629169 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.629539 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.629656 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.629733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.629863 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.629949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 12:51:56.640836 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.639134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.2.81.124:34165)
0924 12:51:56.639332 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.639338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.639462 (+ 124us) server_negotiation.cc:407] Connection header received
0924 12:51:56.639738 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.639743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.639816 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.639898 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.640654 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60636: BlockingRecv error: recv got EOF from 127.0.0.1:60636 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 12:51:56.655593 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.649862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.2.81.125:40459)
0924 12:51:56.650181 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.650189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.650702 (+ 513us) server_negotiation.cc:407] Connection header received
0924 12:51:56.650888 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.650894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.650968 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.651044 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.651456 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.652278 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.653706 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.654305 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.654447 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.654858 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.654977 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.655054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.655197 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.655282 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
W20260924 12:51:56.664710 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.662972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.2.81.124:34165)
0924 12:51:56.663188 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.663194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.663215 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:51:56.663447 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.663453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.663549 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.663630 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.664508 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60650: BlockingRecv error: recv got EOF from 127.0.0.1:60650 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 12:51:56.682169 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.676416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.2.81.125:40459)
0924 12:51:56.676626 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.676633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.677115 (+ 482us) server_negotiation.cc:407] Connection header received
0924 12:51:56.677333 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.677337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.677412 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.677489 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.677927 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.678895 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.680370 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.680839 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.680953 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.681407 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.681515 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.681589 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.681735 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.681836 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 12:51:56.703197 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.698827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.2.81.124:34165)
0924 12:51:56.699043 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.699050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.701679 (+ 2629us) server_negotiation.cc:407] Connection header received
0924 12:51:56.701848 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.701855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.701964 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.702097 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.702964 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60660: BlockingRecv error: recv got EOF from 127.0.0.1:60660 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:51:56.728506 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.722326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.2.81.125:40459)
0924 12:51:56.722521 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.722527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.722810 (+ 283us) server_negotiation.cc:407] Connection header received
0924 12:51:56.722969 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.722975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.723068 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.723181 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.723627 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.724828 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.726361 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.727011 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.727143 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.727686 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.727825 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.727911 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.728119 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.728214 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 12:51:56.736253 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.734333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.2.81.124:34165)
0924 12:51:56.734528 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.734534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.734887 (+ 353us) server_negotiation.cc:407] Connection header received
0924 12:51:56.735050 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.735056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.735160 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.735269 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.736008 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60674: BlockingRecv error: recv got EOF from 127.0.0.1:60674 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 12:51:56.777153 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.771759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.2.81.125:40459)
0924 12:51:56.771986 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.771992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.772197 (+ 205us) server_negotiation.cc:407] Connection header received
0924 12:51:56.772442 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.772447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.772522 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.772651 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.773080 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.773856 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.775243 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.775719 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.775823 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.776308 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.776430 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.776509 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.776718 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.776799 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 12:51:56.785570 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.783445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.2.81.124:34165)
0924 12:51:56.783647 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.783654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.784002 (+ 348us) server_negotiation.cc:407] Connection header received
0924 12:51:56.784200 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.784206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.784291 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.784374 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.785338 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60682: BlockingRecv error: recv got EOF from 127.0.0.1:60682 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 12:51:56.859203 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.853079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.2.81.125:40459)
0924 12:51:56.853297 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.853304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.853707 (+ 403us) server_negotiation.cc:407] Connection header received
0924 12:51:56.853918 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.853923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.853997 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.854143 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:56.854792 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.855547 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.857255 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:56.857733 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:56.857868 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:56.858409 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:56.858544 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:56.858669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:56.858854 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 12:51:56.858932 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 12:51:56.867614 24991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:51:56.865501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.2.81.124:34165)
0924 12:51:56.865789 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.865796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.866139 (+ 343us) server_negotiation.cc:407] Connection header received
0924 12:51:56.866346 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:56.866351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:56.866437 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:56.866523 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:56.867349 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60688: BlockingRecv error: recv got EOF from 127.0.0.1:60688 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 12:51:56.943385 25006 catalog_manager.cc:1259] Loaded cluster ID: d69623a129dd43b288621aad2e701af1
I20260924 12:51:56.943514 25006 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: loading cluster ID for follower catalog manager: success
I20260924 12:51:56.946118 25006 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: acquiring CA information for follower catalog manager: success
I20260924 12:51:56.948065 25006 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dae0a0457da24016955ec1b19b954717: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 12:51:57.005149 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:56.999383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.2.81.125:40459)
0924 12:51:56.999691 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:56.999698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:56.999929 (+ 231us) server_negotiation.cc:407] Connection header received
0924 12:51:57.000168 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:57.000173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:57.000242 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:57.000320 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:57.000801 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.001563 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.003190 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.003642 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.003762 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:57.004166 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:57.004284 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:57.004360 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:57.004546 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 12:51:57.004722 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 12:51:57.018231 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:57.012100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.2.81.124:34165)
0924 12:51:57.012401 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:57.012407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:57.012594 (+ 187us) server_negotiation.cc:407] Connection header received
0924 12:51:57.012827 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:57.012832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:57.013118 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:57.013226 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:57.013752 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.014525 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.015829 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.016256 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.016357 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:57.016919 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:57.017564 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:57.017691 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:57.017848 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 12:51:57.017932 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260924 12:51:57.032055 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:57.025303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.2.81.125:40459)
0924 12:51:57.025528 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:57.025536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:57.026061 (+ 525us) server_negotiation.cc:407] Connection header received
0924 12:51:57.026300 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:57.026306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:57.026400 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:57.026506 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:57.027310 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.028331 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.030221 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.030669 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.030784 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:57.031215 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:57.031362 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:57.031447 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:57.031578 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 12:51:57.031775 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 12:51:57.034035 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:57.026900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.2.81.124:34165)
0924 12:51:57.027096 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:57.027102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:57.027586 (+ 484us) server_negotiation.cc:407] Connection header received
0924 12:51:57.027799 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:57.027804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:57.027878 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:57.027949 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:57.028542 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.029629 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.031406 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.032506 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.032624 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:57.033245 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:57.033376 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:57.033472 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:57.033647 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 12:51:57.033739 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 12:51:57.034655 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:57.025177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.2.81.126:43331)
0924 12:51:57.026162 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:57.026167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:57.026639 (+ 472us) server_negotiation.cc:407] Connection header received
0924 12:51:57.026959 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:57.026965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:57.027197 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:57.027348 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:51:57.028125 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.029008 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.031223 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:57.031942 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:57.032075 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:57.033424 (+ 1349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:51:57.033930 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:51:57.034050 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:57.034099 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 12:51:57.034202 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":737,"threads_started":1}
I20260924 12:51:57.039588 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24773
I20260924 12:51:57.040875 24835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:57.164278 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24773
W20260924 12:51:57.181046 24940 connection.cc:535] server connection from 127.0.0.1:60584 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 12:51:57.181381 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24847
I20260924 12:51:57.182449 24909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:57.294102 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24847
I20260924 12:51:57.311349 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 24921
I20260924 12:51:57.312744 24983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:51:57.425565 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 24921
2026-09-24T12:51:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4394 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 12:51:57 dist-test-slave-9tm1 krb5kdc[25168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 12:51:57 dist-test-slave-9tm1 krb5kdc[25168](info): set up 2 sockets
Sep 24 12:51:57 dist-test-slave-9tm1 krb5kdc[25168](info): commencing operation
krb5kdc: starting...
W20260924 12:51:58.473879 2373 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/test-user.keytab.
Sep 24 12:51:58 dist-test-slave-9tm1 krb5kdc[25168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T12:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T12:51:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.81.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
Entry for principal kudu/127.2.81.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab.
I20260924 12:51:58.744712 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39737
--webserver_interface=127.2.81.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39737,127.2.81.125:36455,127.2.81.124:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 12:51:59.096086 25181 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:51:59.096441 25181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:59.096550 25181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:59.096666 25181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:59.104357 25181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:59.104472 25181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:59.104540 25181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:59.104610 25181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:59.104696 25181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:59.115051 25181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39737,127.2.81.125:36455,127.2.81.124:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.126.keytab
--principal=kudu/127.2.81.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.126:39737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.81.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:59.116442 25181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:59.117813 25181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:59.130287 25186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:59.130314 25187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:59.130352 25189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:59.130967 25181 server_base.cc:1034] running on GCE node
I20260924 12:51:59.132019 25181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:59.133888 25181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:59.135254 25181 hybrid_clock.cc:648] HybridClock initialized: now 1790254319135112 us; error 52 us; skew 500 ppm
I20260924 12:51:59.135591 25181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:59 dist-test-slave-9tm1 krb5kdc[25168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:59.139593 25181 init.cc:377] Logged in from keytab as kudu/127.2.81.126@KRBTEST.COM (short username kudu)
I20260924 12:51:59.141741 25181 webserver.cc:460] Webserver started at http://127.2.81.126:36229/ using document root <none> and password file <none>
I20260924 12:51:59.142580 25181 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:59.142697 25181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:59.143041 25181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:59.146209 25181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data/instance:
uuid: "0e5dcf42b7e94a109d3eab9c36cf985c"
format_stamp: "Formatted at 2026-09-24 12:51:59 on dist-test-slave-9tm1"
server_key: "02a4194e9bb91e61ed953ef0a527dae5"
server_key_iv: "81bfc4c86275f69a95ebb9364a788510"
server_key_version: "encryptionkey@0"
I20260924 12:51:59.147033 25181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal/instance:
uuid: "0e5dcf42b7e94a109d3eab9c36cf985c"
format_stamp: "Formatted at 2026-09-24 12:51:59 on dist-test-slave-9tm1"
server_key: "02a4194e9bb91e61ed953ef0a527dae5"
server_key_iv: "81bfc4c86275f69a95ebb9364a788510"
server_key_version: "encryptionkey@0"
I20260924 12:51:59.152637 25181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 12:51:59.157384 25196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:59.159251 25181 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 12:51:59.159479 25181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal
uuid: "0e5dcf42b7e94a109d3eab9c36cf985c"
format_stamp: "Formatted at 2026-09-24 12:51:59 on dist-test-slave-9tm1"
server_key: "02a4194e9bb91e61ed953ef0a527dae5"
server_key_iv: "81bfc4c86275f69a95ebb9364a788510"
server_key_version: "encryptionkey@0"
I20260924 12:51:59.159727 25181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:59.179062 25181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:59.184223 25181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:59.184543 25181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:59.205672 25181 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.126:39737
I20260924 12:51:59.205700 25247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.126:39737 every 8 connection(s)
I20260924 12:51:59.207104 25181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/data/info.pb
I20260924 12:51:59.211221 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:59.211952 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 25181
I20260924 12:51:59.212127 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-0/wal/instance
I20260924 12:51:59.212534 2373 external_mini_cluster.cc:1428] Setting key 288e3364b193344bc7bf14da8f0df0cf
I20260924 12:51:59.218343 25248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:59 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.81.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.125@KRBTEST.COM" created.
I20260924 12:51:59.244295 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.222062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39737 (local address 127.0.0.1:49724)
0924 12:51:59.223214 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.223267 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:59.225583 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:59.226631 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:59.226637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:59.227406 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:59.227879 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.227903 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.229583 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.229589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.231214 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.231231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.232284 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.232294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.232461 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.233259 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:59.233293 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:59.236181 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:59.239777 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.239789 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.239799 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.240418 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.241047 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.241051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.241053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.241247 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.242271 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:59.242277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:59.242483 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 12:51:59.242695 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 12:51:59.243110 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"spinlock_wait_cycles":68352,"thread_start_us":685,"threads_started":1}
I20260924 12:51:59.244346 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.222063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.2.81.126:39737)
0924 12:51:59.222961 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:59.222981 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:59.223402 (+ 421us) server_negotiation.cc:407] Connection header received
0924 12:51:59.226088 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:59.226095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:59.226390 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:59.226577 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:59.228131 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.229407 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.231402 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.232009 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.232167 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.236490 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:59.236506 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:59.236524 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:59.236563 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:59.239597 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.240636 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.240646 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.240650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.240892 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.241350 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.241356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.241359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.242116 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:59.242303 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:59.242973 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 12:51:59.243238 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":373,"threads_started":1}
I20260924 12:51:59.247597 25248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:51:59.248777 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.248329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36670)
0924 12:51:59.248522 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.248644 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 12:51:59.249100 25197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
I20260924 12:51:59.250880 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.250403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36686)
0924 12:51:59.250583 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.250682 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 12:51:59.251148 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:36455: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab.
Entry for principal kudu/127.2.81.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab.
I20260924 12:51:59.277647 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:36455
--webserver_interface=127.2.81.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab
--principal=kudu/127.2.81.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39737,127.2.81.125:36455,127.2.81.124:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:51:59.300578 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } attempt: 1
I20260924 12:51:59.301831 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.301351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36690)
0924 12:51:59.301575 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.301696 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 12:51:59.302858 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.302465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36706)
0924 12:51:59.302610 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.302718 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 12:51:59.303049 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:36455: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
I20260924 12:51:59.371335 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } attempt: 2
I20260924 12:51:59.372682 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.372077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36722)
0924 12:51:59.372371 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.372477 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 12:51:59.373760 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.373373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36728)
0924 12:51:59.373528 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.373617 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 12:51:59.373987 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:36455: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
I20260924 12:51:59.465270 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } attempt: 3
I20260924 12:51:59.466480 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.466036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36732)
0924 12:51:59.466236 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.466333 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 12:51:59.467473 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.467108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36736)
0924 12:51:59.467253 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.467329 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 12:51:59.467651 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:36455: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
I20260924 12:51:59.610930 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } attempt: 4
I20260924 12:51:59.612195 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.611630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36738)
0924 12:51:59.611975 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.612061 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260924 12:51:59.613278 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.612877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36744)
0924 12:51:59.613059 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.613148 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 12:51:59.613469 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.125:36455: Network error: Client connection negotiation failed: client connection to 127.2.81.125:36455: connect: Connection refused (error 111)
W20260924 12:51:59.630259 25255 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 12:51:59.630645 25255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:51:59.630744 25255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:51:59.630863 25255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 12:51:59.639009 25255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:51:59.639151 25255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:51:59.639238 25255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:51:59.639304 25255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:51:59.639391 25255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:51:59.650243 25255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39737,127.2.81.125:36455,127.2.81.124:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.125.keytab
--principal=kudu/127.2.81.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.125:36455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.81.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:51:59.651628 25255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:51:59.653021 25255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:51:59.665532 25261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:59.665617 25260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:51:59.665654 25263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:51:59.666303 25255 server_base.cc:1034] running on GCE node
I20260924 12:51:59.667138 25255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:51:59.668681 25255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:51:59.669950 25255 hybrid_clock.cc:648] HybridClock initialized: now 1790254319669880 us; error 37 us; skew 500 ppm
I20260924 12:51:59.670320 25255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:51:59 dist-test-slave-9tm1 krb5kdc[25168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:51:59.673631 25255 init.cc:377] Logged in from keytab as kudu/127.2.81.125@KRBTEST.COM (short username kudu)
I20260924 12:51:59.675231 25255 webserver.cc:460] Webserver started at http://127.2.81.125:42975/ using document root <none> and password file <none>
I20260924 12:51:59.675925 25255 fs_manager.cc:359] Metadata directory not provided
I20260924 12:51:59.676011 25255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:51:59.676283 25255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:51:59.679899 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data/instance:
uuid: "838b9460480842c2a35a6504ea654687"
format_stamp: "Formatted at 2026-09-24 12:51:59 on dist-test-slave-9tm1"
server_key: "d33f5caddce13f81fbf3c567a9d53263"
server_key_iv: "91deb9ead8a2108aff65350a572b1763"
server_key_version: "encryptionkey@0"
I20260924 12:51:59.680838 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal/instance:
uuid: "838b9460480842c2a35a6504ea654687"
format_stamp: "Formatted at 2026-09-24 12:51:59 on dist-test-slave-9tm1"
server_key: "d33f5caddce13f81fbf3c567a9d53263"
server_key_iv: "91deb9ead8a2108aff65350a572b1763"
server_key_version: "encryptionkey@0"
I20260924 12:51:59.686594 25255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 12:51:59.690485 25270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:51:59.692255 25255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 12:51:59.692448 25255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal
uuid: "838b9460480842c2a35a6504ea654687"
format_stamp: "Formatted at 2026-09-24 12:51:59 on dist-test-slave-9tm1"
server_key: "d33f5caddce13f81fbf3c567a9d53263"
server_key_iv: "91deb9ead8a2108aff65350a572b1763"
server_key_version: "encryptionkey@0"
I20260924 12:51:59.692804 25255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:51:59.715016 25255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:51:59.719759 25255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:51:59.720029 25255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:51:59.738173 25255 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.125:36455
I20260924 12:51:59.738193 25321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.125:36455 every 8 connection(s)
I20260924 12:51:59.739565 25255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/data/info.pb
I20260924 12:51:59.743449 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:51:59.744148 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 25255
I20260924 12:51:59.744320 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-1/wal/instance
I20260924 12:51:59.744869 2373 external_mini_cluster.cc:1428] Setting key f9157687f6cb15abd1d9ef4d83ff1849
I20260924 12:51:59.753108 25322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.81.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.81.124@KRBTEST.COM" created.
Sep 24 12:51:59 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:51:59.782995 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.756571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.2.81.126:39737)
0924 12:51:59.757104 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:59.757109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:59.758458 (+ 1349us) server_negotiation.cc:407] Connection header received
0924 12:51:59.760520 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:59.760527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:59.760616 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:59.760713 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:59.762468 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.763185 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.765051 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.765499 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.765624 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.775092 (+ 9468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:59.775101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:59.775106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:59.775167 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:59.778701 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.779828 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.779834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.779837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.780036 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.780709 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.780714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.780717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.781291 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:59.781461 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:59.782358 (+ 897us) server_negotiation.cc:299] Negotiation successful
0924 12:51:59.782617 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260924 12:51:59.784714 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.757382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39737 (local address 127.0.0.1:49736)
0924 12:51:59.758263 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.758314 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:59.760202 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:59.761012 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:59.761025 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:59.761628 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:59.762214 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.762234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.763364 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.763368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.764824 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.764840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.765745 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.765753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.765998 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.770319 (+ 4321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:59.770350 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:59.774735 (+ 4385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:59.779052 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.779061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.779080 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.779658 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.780329 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.780333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.780336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.780571 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.781485 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:59.781494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:59.781915 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 12:51:59.782267 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 12:51:59.783041 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":373,"threads_started":1}
I20260924 12:51:59.787478 25322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:51:59 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:51:59.804296 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.788326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36758)
0924 12:51:59.788539 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.788559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:59.788690 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:59.789970 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:59.789975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:59.790457 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:59.790671 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.790683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.792161 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.792166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.793409 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.793426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.794470 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.794476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.794647 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.795288 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:59.795312 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:59.798078 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:59.801877 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.801883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.801888 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.802241 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.802605 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.802610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.802613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.802738 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.803338 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:59.803342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:59.803473 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 12:51:59.803596 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 12:51:59.803900 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 12:51:59.804404 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.788591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.2.81.125:36455)
0924 12:51:59.789440 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:59.789447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:59.789477 (+ 30us) server_negotiation.cc:407] Connection header received
0924 12:51:59.789532 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:59.789538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:59.789767 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:59.789908 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:59.790887 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.792004 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.793638 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.794303 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.794417 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.798310 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:59.798323 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:59.798338 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:59.798371 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:59.801728 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.802368 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.802375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.802377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.802495 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.802834 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.802838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.802840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.803191 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:59.803399 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:59.803759 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 12:51:59.804023 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":660,"thread_start_us":505,"threads_started":1}
I20260924 12:51:59.806890 25322 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.81.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab.
Entry for principal kudu/127.2.81.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab.
I20260924 12:51:59.808116 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.807597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49384)
0924 12:51:59.807805 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.807950 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 12:51:59.808336 25271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:51:59.808844 2373 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYQebFB/build/asan/bin/kudu
/tmp/dist-test-taskYQebFB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:46103
--webserver_interface=127.2.81.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab
--principal=kudu/127.2.81.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.81.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.81.126:39737,127.2.81.125:36455,127.2.81.124:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 12:51:59.809945 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.809599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49386)
0924 12:51:59.809751 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.809831 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 12:51:59.810141 25322 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:51:59.859477 25322 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 1
I20260924 12:51:59.860780 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.860146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49390)
0924 12:51:59.860400 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.860561 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 12:51:59.862047 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.861558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49394)
0924 12:51:59.861754 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.861887 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 12:51:59.862320 25322 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:51:59.912828 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } attempt: 5
Sep 24 12:51:59 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:51:59.929865 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.913784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36760)
0924 12:51:59.913977 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.913998 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 12:51:59.914110 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:51:59.914591 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:51:59.914596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:51:59.914991 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:51:59.915181 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.915192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.916212 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.916217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.917476 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:51:59.917496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.918841 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.918859 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:51:59.919148 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.919869 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:51:59.919889 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:51:59.922409 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:51:59.926704 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.926708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.926711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.927158 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.927787 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:51:59.927791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:51:59.927794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:51:59.928042 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:51:59.928951 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:51:59.928955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:51:59.929196 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 12:51:59.929328 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 12:51:59.929606 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 12:51:59.930042 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.913990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.2.81.125:36455)
0924 12:51:59.914291 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 12:51:59.914296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:51:59.914311 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:51:59.914377 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:51:59.914381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:51:59.914441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:51:59.914525 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:51:59.915341 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.916064 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.917697 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:51:59.918347 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:51:59.918488 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:51:59.922679 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:51:59.922688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:51:59.922692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:51:59.922738 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:51:59.926472 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.927388 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.927395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.927399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.927600 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:51:59.928224 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:51:59.928231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:51:59.928235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:51:59.928720 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:51:59.928910 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 12:51:59.929496 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 12:51:59.929754 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 12:51:59.930609 25322 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 2
I20260924 12:51:59.931288 25248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } has no permanent_uuid. Determining permanent_uuid...
I20260924 12:51:59.931936 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.931475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49398)
0924 12:51:59.931657 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.931760 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 12:51:59.932350 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.931915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49414)
0924 12:51:59.932084 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.932174 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 12:51:59.933070 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.932641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49426)
0924 12:51:59.932814 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.932894 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 12:51:59.933297 25322 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:51:59.933405 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.932966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49428)
0924 12:51:59.933201 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.933304 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260924 12:51:59.933615 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:51:59.984861 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 1
I20260924 12:51:59.986191 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.985666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49430)
0924 12:51:59.985882 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.985978 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 12:51:59.987387 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:51:59.986971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49434)
0924 12:51:59.987150 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 12:51:59.987264 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 12:51:59.987612 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:52:00.024551 25322 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 3
I20260924 12:52:00.025943 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.025376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49440)
0924 12:52:00.025618 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.025731 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 12:52:00.027917 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.026936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49448)
0924 12:52:00.027587 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.027726 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260924 12:52:00.028263 25322 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:52:00.055887 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 2
I20260924 12:52:00.057451 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.056683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49456)
0924 12:52:00.057089 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.057201 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 12:52:00.058583 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.058210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49468)
0924 12:52:00.058365 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.058446 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 12:52:00.058797 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
I20260924 12:52:00.165019 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 3
I20260924 12:52:00.166226 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.165675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49482)
0924 12:52:00.165946 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.166037 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260924 12:52:00.166989 25330 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260924 12:52:00.167332 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.166899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49484)
0924 12:52:00.167113 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.167194 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 12:52:00.167366 25330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 12:52:00.167455 25330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 12:52:00.167531 25248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
W20260924 12:52:00.167577 25330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 12:52:00.171559 25322 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 4
I20260924 12:52:00.172823 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.172297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49500)
0924 12:52:00.172531 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.172667 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 12:52:00.173823 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.173487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49514)
0924 12:52:00.173649 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.173718 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 12:52:00.174019 25322 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.81.124:46103: Network error: Client connection negotiation failed: client connection to 127.2.81.124:46103: connect: Connection refused (error 111)
W20260924 12:52:00.175829 25330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 12:52:00.175930 25330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 12:52:00.176023 25330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 12:52:00.176088 25330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 12:52:00.176170 25330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 12:52:00.186926 25330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.81.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.81.126:39737,127.2.81.125:36455,127.2.81.124:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/krb5kdc/kudu_127.2.81.124.keytab
--principal=kudu/127.2.81.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.81.124:46103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.81.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 11:23:59 UTC on ccycloud.quasar-uwcnyx.root.comops.site
build id 4669763
ASAN enabled
I20260924 12:52:00.188298 25330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 12:52:00.189693 25330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 12:52:00.202322 25338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:52:00.202299 25336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 12:52:00.202564 25335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 12:52:00.203372 25330 server_base.cc:1034] running on GCE node
I20260924 12:52:00.204165 25330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 12:52:00.205915 25330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 12:52:00.207183 25330 hybrid_clock.cc:648] HybridClock initialized: now 1790254320207071 us; error 78 us; skew 500 ppm
I20260924 12:52:00.207576 25330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 12:52:00.212257 25330 init.cc:377] Logged in from keytab as kudu/127.2.81.124@KRBTEST.COM (short username kudu)
I20260924 12:52:00.214274 25330 webserver.cc:460] Webserver started at http://127.2.81.124:35757/ using document root <none> and password file <none>
I20260924 12:52:00.215204 25330 fs_manager.cc:359] Metadata directory not provided
I20260924 12:52:00.215332 25330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 12:52:00.215694 25330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 12:52:00.218818 25330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data/instance:
uuid: "73e7f333920a4003a85eeb87a7dd73f6"
format_stamp: "Formatted at 2026-09-24 12:52:00 on dist-test-slave-9tm1"
server_key: "675f97361fd5af1fb580a72c8902a1b6"
server_key_iv: "af03743faa4db226a4f1a2ee3b842c46"
server_key_version: "encryptionkey@0"
I20260924 12:52:00.219588 25330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal/instance:
uuid: "73e7f333920a4003a85eeb87a7dd73f6"
format_stamp: "Formatted at 2026-09-24 12:52:00 on dist-test-slave-9tm1"
server_key: "675f97361fd5af1fb580a72c8902a1b6"
server_key_iv: "af03743faa4db226a4f1a2ee3b842c46"
server_key_version: "encryptionkey@0"
I20260924 12:52:00.225677 25330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 12:52:00.229516 25345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 12:52:00.231161 25330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 12:52:00.231338 25330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data,/tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal
uuid: "73e7f333920a4003a85eeb87a7dd73f6"
format_stamp: "Formatted at 2026-09-24 12:52:00 on dist-test-slave-9tm1"
server_key: "675f97361fd5af1fb580a72c8902a1b6"
server_key_iv: "af03743faa4db226a4f1a2ee3b842c46"
server_key_version: "encryptionkey@0"
I20260924 12:52:00.231546 25330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 12:52:00.252542 25330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 12:52:00.257733 25330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 12:52:00.257984 25330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 12:52:00.276950 25330 rpc_server.cc:307] RPC server started. Bound to: 127.2.81.124:46103
I20260924 12:52:00.276957 25396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.81.124:46103 every 8 connection(s)
I20260924 12:52:00.278395 25330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/data/info.pb
I20260924 12:52:00.283557 25397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 12:52:00.286669 2373 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYQebFB/build/asan/bin/kudu as pid 25330
I20260924 12:52:00.286972 2373 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYQebFB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790253949676453-2373-0/minicluster-data/master-2/wal/instance
I20260924 12:52:00.287410 2373 external_mini_cluster.cc:1428] Setting key 4d75bd1c35ff85359faa8d06a3288b9c
I20260924 12:52:00.292033 25397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:52:00.313251 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.290152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.2.81.124:46103)
0924 12:52:00.291020 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.291036 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.291101 (+ 65us) server_negotiation.cc:407] Connection header received
0924 12:52:00.293227 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.293262 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.293942 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.294457 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.295862 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.297877 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.299667 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.300608 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.300798 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.305371 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.305382 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.305397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.305443 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.308655 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.309508 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.309517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.309529 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.309739 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.310128 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.310133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.310136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.310914 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.311243 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.311669 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.312106 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":376,"threads_started":1}
I20260924 12:52:00.314013 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.296373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.2.81.126:39737)
0924 12:52:00.297271 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.297278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.298177 (+ 899us) server_negotiation.cc:407] Connection header received
0924 12:52:00.298419 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.298424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.298496 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.298584 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.300223 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.301214 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.303093 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.303524 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.303639 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.307426 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.307434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.307439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.307492 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.310821 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.311584 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.311589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.311591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.311821 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.312395 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.312399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.312401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.312679 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.312833 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.313344 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.313598 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":438,"threads_started":1}
I20260924 12:52:00.314039 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.296315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.126:39737 (local address 127.0.0.1:49746)
0924 12:52:00.297977 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.298033 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 12:52:00.298266 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:52:00.298691 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:52:00.298700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:52:00.299625 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:52:00.300025 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.300040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.301363 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.301367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.302899 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.302917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.303783 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.303791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.304057 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.304740 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:52:00.304765 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:52:00.307195 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:52:00.310972 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.310981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.310987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.311434 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.312111 (+ 677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.312115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.312117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.312273 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.312816 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:52:00.312824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:52:00.313035 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 12:52:00.313221 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 12:52:00.313600 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1447,"thread_start_us":469,"threads_started":1}
I20260924 12:52:00.316329 25397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.126@KRBTEST.COM
I20260924 12:52:00.331023 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.317169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.2.81.125:36455)
0924 12:52:00.317332 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.317338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.317503 (+ 165us) server_negotiation.cc:407] Connection header received
0924 12:52:00.317655 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.317659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.317719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.317800 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.318620 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.319298 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.320478 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.320897 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.321034 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.324566 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.324572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.324575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.324616 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.328039 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.328791 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.328797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.328799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.329084 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.329523 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.329529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.329533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.329925 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.330041 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.330518 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.330747 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 12:52:00.331367 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.317152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.125:36455 (local address 127.0.0.1:36764)
0924 12:52:00.317411 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.317425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 12:52:00.317538 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:52:00.317839 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:52:00.317843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:52:00.318242 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:52:00.318429 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.318438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.319433 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.319439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.320328 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.320340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.321078 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.321085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.321274 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.321850 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:52:00.321870 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:52:00.324307 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:52:00.328237 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.328243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.328247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.328658 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.329226 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.329230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.329232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.329402 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.330056 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:52:00.330062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:52:00.330244 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 12:52:00.330387 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 12:52:00.330747 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 12:52:00.332165 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.317219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.2.81.126:39737)
0924 12:52:00.317490 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.317496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.317513 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:52:00.317569 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.317574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.317654 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.317746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.319010 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.319712 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.320935 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.321438 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.321556 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.326098 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.326105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.326110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.326159 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.329033 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.329931 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.329936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.329938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.330161 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.330582 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.330586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.330589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.330938 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.331238 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.331619 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.331868 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 12:52:00.332965 25397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.124@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.81.125@KRBTEST.COM
I20260924 12:52:00.344753 25248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 4
I20260924 12:52:00.349632 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.333747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49534)
0924 12:52:00.333936 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.333956 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:52:00.334106 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:52:00.334480 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:52:00.334486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:52:00.334923 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:52:00.335221 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.335266 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.336552 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.336558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.338375 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.338390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.339513 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.339518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.339767 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.340410 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:52:00.340440 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:52:00.342587 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:52:00.346572 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.346577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.346581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.347101 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.347452 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.347457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.347461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.347583 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.348267 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:52:00.348273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:52:00.348467 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 12:52:00.348794 (+ 327us) client_negotiation.cc:240] Negotiation successful
0924 12:52:00.349202 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 12:52:00.349910 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.333959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.2.81.124:46103)
0924 12:52:00.334126 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.334130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.334145 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:52:00.334262 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.334267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.334344 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.334439 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.335499 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.336430 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.338558 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.339330 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.339482 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.342824 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.342832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.342837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.342878 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.346444 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.347222 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.347225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.347227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.347344 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.347738 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.347743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.347745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.348145 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.348291 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.348880 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.349382 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":243,"server-negotiator.queue_time_us":67}
I20260924 12:52:00.353982 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.333972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.2.81.125:36455)
0924 12:52:00.334271 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.334276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.334291 (+ 15us) server_negotiation.cc:407] Connection header received
0924 12:52:00.334345 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.334349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.334413 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.334492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.336166 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.337670 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.339740 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.340392 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.340534 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.345712 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.345721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.345726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.345779 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.350389 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.351279 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.351286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.351289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.351474 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.352153 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.352159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.352162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.352719 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.353061 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.353391 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.353660 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.126@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
I20260924 12:52:00.356441 25397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: Bootstrap starting.
I20260924 12:52:00.359423 2373 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 12:52:00.360100 25397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: Neither blocks nor log segments found. Creating new log.
I20260924 12:52:00.361462 25397 log.cc:824] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: Log is configured to *not* fsync() on all Append() calls
I20260924 12:52:00.362941 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.345377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49548)
0924 12:52:00.345789 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.345814 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 12:52:00.345940 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:52:00.346769 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:52:00.346775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:52:00.347299 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:52:00.347522 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.347536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.349185 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.349192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.350549 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.350572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.351557 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.351564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.351793 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.352542 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:52:00.352567 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:52:00.355349 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:52:00.359798 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.359804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.359808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.360377 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.360818 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.360824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.360827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.361045 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.361792 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:52:00.361797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:52:00.361996 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 12:52:00.362166 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 12:52:00.362514 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 12:52:00.364059 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.345561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.2.81.124:46103)
0924 12:52:00.346441 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.346446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.346468 (+ 22us) server_negotiation.cc:407] Connection header received
0924 12:52:00.346518 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.346524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.346598 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.346711 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.347702 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.349026 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.350749 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.351385 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.351509 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.355737 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.355750 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.355756 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.355803 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.359648 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.360506 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.360512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.360516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.360694 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.361153 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.361157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.361160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.361554 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.361718 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.363396 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.363630 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":590,"threads_started":1}
I20260924 12:52:00.364532 25397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: No bootstrap required, opened a new log
I20260924 12:52:00.362118 25223 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758
I20260924 12:52:00.370422 25248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c: Bootstrap starting.
I20260924 12:52:00.375053 25248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c: Neither blocks nor log segments found. Creating new log.
I20260924 12:52:00.375672 25397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.376139 25397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e7f333920a4003a85eeb87a7dd73f6, State: Initialized, Role: FOLLOWER
I20260924 12:52:00.376485 25248 log.cc:824] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c: Log is configured to *not* fsync() on all Append() calls
I20260924 12:52:00.376667 25397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.377954 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.378455 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.378793 25397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:52:00.380107 25248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c: No bootstrap required, opened a new log
I20260924 12:52:00.388485 25248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.388960 25248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5dcf42b7e94a109d3eab9c36cf985c, State: Initialized, Role: FOLLOWER
W20260924 12:52:00.389127 25417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 12:52:00.389307 25417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 12:52:00.389526 25248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.390746 25419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.391453 25419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.391805 25248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:52:00.391933 25223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 12:52:00.392124 25223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:52:00.395596 25223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.397569 25223 leader_election.cc:288] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [CANDIDATE]: Term 1 election: Requested vote from peers 838b9460480842c2a35a6504ea654687 (127.2.81.125:36455), 73e7f333920a4003a85eeb87a7dd73f6 (127.2.81.124:46103)
I20260924 12:52:00.386861 25330 master_runner.cc:186] Error getting master registration for 127.2.81.126:39737: OK, instance_id { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" instance_seqno: 1790254319200123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 12:52:00.398402 25330 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 12:52:00.398582 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73e7f333920a4003a85eeb87a7dd73f6"
I20260924 12:52:00.398938 25372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:52:00.402258 25372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5dcf42b7e94a109d3eab9c36cf985c in term 1.
I20260924 12:52:00.403061 25197 leader_election.cc:302] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5dcf42b7e94a109d3eab9c36cf985c, 73e7f333920a4003a85eeb87a7dd73f6; no voters:
I20260924 12:52:00.403712 25419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 12:52:00.423415 25419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [term 1 LEADER]: Becoming Leader. State: Replica: 0e5dcf42b7e94a109d3eab9c36cf985c, State: Running, Role: LEADER
I20260924 12:52:00.424229 25419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.434140 25427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5dcf42b7e94a109d3eab9c36cf985c. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.434621 25427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: This master's current role is: LEADER
W20260924 12:52:00.437745 25202 tablet.cc:2369] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:52:00.440716 25433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 12:52:00.449996 25433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 12:52:00.466657 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.409897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.2.81.124:46103)
0924 12:52:00.410159 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.410165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.419405 (+ 9240us) server_negotiation.cc:407] Connection header received
0924 12:52:00.419679 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.419686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.419803 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.419882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.422119 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.423211 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.431644 (+ 8433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.432347 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.432462 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.444524 (+ 12062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.444533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.444539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.444600 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.449282 (+ 4682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.450410 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.450418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.450422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.450627 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.464522 (+ 13895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.464530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.464533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.465040 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.465378 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.465970 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.466247 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 12:52:00.469887 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.403335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.2.81.126:39737)
0924 12:52:00.403702 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.403709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.409611 (+ 5902us) server_negotiation.cc:407] Connection header received
0924 12:52:00.413516 (+ 3905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.413524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.413620 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.413705 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.426837 (+ 13132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.428009 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.436017 (+ 8008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.436757 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.436875 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.451498 (+ 14623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.451546 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.451552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.451610 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.458081 (+ 6471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.464624 (+ 6543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.464630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.464633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.464835 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.467493 (+ 2658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.467500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.467504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.467930 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.468207 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.469227 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.469502 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260924 12:52:00.470870 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.403509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.2.81.125:36455)
0924 12:52:00.403835 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.403841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.412773 (+ 8932us) server_negotiation.cc:407] Connection header received
0924 12:52:00.413009 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.413016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.413116 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.413211 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.425344 (+ 12133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.426434 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.438131 (+ 11697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.438816 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.438929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.459553 (+ 20624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.459562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.459568 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.459632 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.464086 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.466206 (+ 2120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.466214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.466217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.466412 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.466894 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.466898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.466901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.467299 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.469623 (+ 2324us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.470233 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.470511 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260924 12:52:00.473196 25322 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } attempt: 5
I20260924 12:52:00.484411 25372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [term 1 FOLLOWER]: Refusing update from remote peer 0e5dcf42b7e94a109d3eab9c36cf985c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Sep 24 12:52:00 dist-test-slave-9tm1 krb5kdc[25168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790254319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.81.125@KRBTEST.COM for kudu/127.2.81.124@KRBTEST.COM
I20260924 12:52:00.485773 25427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:52:00.506409 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.473917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.81.124:46103 (local address 127.0.0.1:49566)
0924 12:52:00.474322 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 12:52:00.474342 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 12:52:00.474453 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 12:52:00.475536 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 12:52:00.475540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 12:52:00.476064 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 12:52:00.476290 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.476303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.477844 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.477849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.479237 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 12:52:00.479253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.481052 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.481061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 12:52:00.481287 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.482151 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 12:52:00.482199 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 12:52:00.486899 (+ 4700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 12:52:00.498176 (+ 11277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.498183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.498187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.499196 (+ 1009us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.501089 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 12:52:00.501097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 12:52:00.501101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 12:52:00.501480 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 12:52:00.504592 (+ 3112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 12:52:00.504600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 12:52:00.504946 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 12:52:00.505268 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 12:52:00.505831 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291}
W20260924 12:52:00.510102 25351 tablet.cc:2369] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 12:52:00.510797 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:00.474813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.2.81.124:46103)
0924 12:52:00.475028 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.475033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.475052 (+ 19us) server_negotiation.cc:407] Connection header received
0924 12:52:00.475123 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.475128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.475213 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.475313 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.476523 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.477623 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.479539 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:00.480764 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:00.481905 (+ 1141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:00.487252 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 12:52:00.487284 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 12:52:00.487289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 12:52:00.487361 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 12:52:00.497690 (+ 10329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.499719 (+ 2029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.499727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.499824 (+ 97us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.500630 (+ 806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 12:52:00.501993 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 12:52:00.502000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 12:52:00.502004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 12:52:00.504024 (+ 2020us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 12:52:00.507583 (+ 3559us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:00.508256 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 12:52:00.509264 (+ 1008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 12:52:00.517990 25322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: Bootstrap starting.
I20260924 12:52:00.524871 25322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: Neither blocks nor log segments found. Creating new log.
I20260924 12:52:00.530122 25322 log.cc:824] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: Log is configured to *not* fsync() on all Append() calls
I20260924 12:52:00.534633 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5dcf42b7e94a109d3eab9c36cf985c. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.535015 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.553753 25436 mvcc.cc:205] Tried to move back new op lower bound from 7332881696671014912 to 7332881696465235968. Current Snapshot: MvccSnapshot[applied={T|T < 7332881696671014912}]
I20260924 12:52:00.566007 25322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: No bootstrap required, opened a new log
I20260924 12:52:00.569106 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.569484 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.569934 25427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.572991 25427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: This master's current role is: LEADER
I20260924 12:52:00.580475 25433 catalog_manager.cc:1347] Generated new cluster ID: 718a659c2cd842e58be4ddb3cd2da491
I20260924 12:52:00.580610 25433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 12:52:00.605976 25322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.606969 25322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838b9460480842c2a35a6504ea654687, State: Initialized, Role: FOLLOWER
I20260924 12:52:00.608171 25322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } }
I20260924 12:52:00.633785 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.634372 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.636067 25433 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 12:52:00.638252 25433 catalog_manager.cc:1504] Loading token signing keys...
I20260924 12:52:00.643114 25297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "838b9460480842c2a35a6504ea654687"
I20260924 12:52:00.643721 25297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [term 0 FOLLOWER]: Advancing to term 1
I20260924 12:52:00.647452 25297 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5dcf42b7e94a109d3eab9c36cf985c in term 1.
I20260924 12:52:00.648594 25296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [term 1 FOLLOWER]: Refusing update from remote peer 0e5dcf42b7e94a109d3eab9c36cf985c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 12:52:00.649843 25322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 12:52:00.650267 25419 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [LEADER]: Connected to new peer: Peer: permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 12:52:00.651937 25448 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 12:52:00.667228 25433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c: Generated new TSK 0
I20260924 12:52:00.668530 25433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 12:52:00.672252 25181 master_runner.cc:186] Error getting master registration for 127.2.81.125:36455: OK, instance_id { permanent_uuid: "838b9460480842c2a35a6504ea654687" instance_seqno: 1790254319732949 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 12:52:00.684883 25181 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 12:52:00.725267 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5dcf42b7e94a109d3eab9c36cf985c. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.725630 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.725587 25427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.725876 25427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5dcf42b7e94a109d3eab9c36cf985c [sys.catalog]: This master's current role is: LEADER
I20260924 12:52:00.726831 25456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e5dcf42b7e94a109d3eab9c36cf985c" member_type: VOTER last_known_addr { host: "127.2.81.126" port: 39737 } } peers { permanent_uuid: "838b9460480842c2a35a6504ea654687" member_type: VOTER last_known_addr { host: "127.2.81.125" port: 36455 } } peers { permanent_uuid: "73e7f333920a4003a85eeb87a7dd73f6" member_type: VOTER last_known_addr { host: "127.2.81.124" port: 46103 } } }
I20260924 12:52:00.729069 25456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 12:52:00.784936 25465 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 12:52:00.807372 2373 mini_kdc.cc:348] Time spent kdestroy: real 0.113s user 0.003s sys 0.021s
W20260924 12:52:00.827836 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.811332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.2.81.125:36455)
0924 12:52:00.811518 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.811524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.825136 (+ 13612us) server_negotiation.cc:407] Connection header received
0924 12:52:00.826179 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.826187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.826290 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.826364 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.827577 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36784: BlockingRecv error: recv got EOF from 127.0.0.1:36784 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 12:52:00.830068 25402 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.811281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.2.81.126:39737)
0924 12:52:00.811512 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.811519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.824571 (+ 13052us) server_negotiation.cc:407] Connection header received
0924 12:52:00.828412 (+ 3841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.828419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.828516 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.828597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.829825 (+ 1228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49782: BlockingRecv error: recv got EOF from 127.0.0.1:49782 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 12:52:00.832758 25403 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.825400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.2.81.124:46103)
0924 12:52:00.825589 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.825598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.830842 (+ 5244us) server_negotiation.cc:407] Connection header received
0924 12:52:00.831418 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.831424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.831506 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.831574 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.832503 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49570: BlockingRecv error: recv got EOF from 127.0.0.1:49570 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 12:52:00.843230 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.838463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.2.81.125:36455)
0924 12:52:00.838654 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.838660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.841582 (+ 2922us) server_negotiation.cc:407] Connection header received
0924 12:52:00.841850 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.841856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.841994 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.842098 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.842965 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36796: BlockingRecv error: recv got EOF from 127.0.0.1:36796 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 12:52:00.850647 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.848298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.2.81.125:36455)
0924 12:52:00.848476 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.848483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.848825 (+ 342us) server_negotiation.cc:407] Connection header received
0924 12:52:00.849078 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.849084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.849176 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.849282 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.850410 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36802: BlockingRecv error: recv got EOF from 127.0.0.1:36802 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 12:52:00.865624 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.863582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.2.81.125:36455)
0924 12:52:00.863753 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.863759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.863840 (+ 81us) server_negotiation.cc:407] Connection header received
0924 12:52:00.864189 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.864196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.864305 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.864434 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.865381 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36812: BlockingRecv error: recv got EOF from 127.0.0.1:36812 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 12:52:00.876432 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.874140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.2.81.125:36455)
0924 12:52:00.874333 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.874339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.874892 (+ 553us) server_negotiation.cc:407] Connection header received
0924 12:52:00.875103 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.875109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.875224 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.875342 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.876203 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36814: BlockingRecv error: recv got EOF from 127.0.0.1:36814 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 12:52:00.893952 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.891485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.2.81.125:36455)
0924 12:52:00.891711 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.891717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.892310 (+ 593us) server_negotiation.cc:407] Connection header received
0924 12:52:00.892533 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.892539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.892641 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.892760 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.893715 (+ 955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36820: BlockingRecv error: recv got EOF from 127.0.0.1:36820 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 12:52:00.916518 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.914145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.2.81.125:36455)
0924 12:52:00.914340 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.914345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.914907 (+ 562us) server_negotiation.cc:407] Connection header received
0924 12:52:00.915129 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.915136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.915240 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.915357 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.916259 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36822: BlockingRecv error: recv got EOF from 127.0.0.1:36822 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 12:52:00.957847 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:00.954951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.2.81.125:36455)
0924 12:52:00.955321 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:00.955329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:00.956143 (+ 814us) server_negotiation.cc:407] Connection header received
0924 12:52:00.956375 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:00.956382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:00.956483 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:00.956610 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:00.957567 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36830: BlockingRecv error: recv got EOF from 127.0.0.1:36830 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 12:52:01.027732 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:01.025572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.2.81.125:36455)
0924 12:52:01.025820 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.025827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.026237 (+ 410us) server_negotiation.cc:407] Connection header received
0924 12:52:01.026461 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.026466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.026545 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.026638 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:01.027502 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36838: BlockingRecv error: recv got EOF from 127.0.0.1:36838 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 12:52:01.161935 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:01.160037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.2.81.125:36455)
0924 12:52:01.160248 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.160253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.160375 (+ 122us) server_negotiation.cc:407] Connection header received
0924 12:52:01.160580 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.160584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.160659 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.160747 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:01.161626 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36848: BlockingRecv error: recv got EOF from 127.0.0.1:36848 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 12:52:01.392081 25417 catalog_manager.cc:1259] Loaded cluster ID: 718a659c2cd842e58be4ddb3cd2da491
I20260924 12:52:01.392277 25417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: loading cluster ID for follower catalog manager: success
I20260924 12:52:01.395049 25417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: acquiring CA information for follower catalog manager: success
I20260924 12:52:01.397197 25417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73e7f333920a4003a85eeb87a7dd73f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 12:52:01.423839 25328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 12:52:01.421806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.2.81.125:36455)
0924 12:52:01.422126 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.422133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.422468 (+ 335us) server_negotiation.cc:407] Connection header received
0924 12:52:01.422681 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.422685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.422772 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.422860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 12:52:01.423623 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36862: BlockingRecv error: recv got EOF from 127.0.0.1:36862 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
I20260924 12:52:01.789450 25465 catalog_manager.cc:1259] Loaded cluster ID: 718a659c2cd842e58be4ddb3cd2da491
I20260924 12:52:01.789601 25465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: loading cluster ID for follower catalog manager: success
I20260924 12:52:01.792733 25465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: acquiring CA information for follower catalog manager: success
I20260924 12:52:01.795517 25465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 838b9460480842c2a35a6504ea654687: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 12:52:01.952720 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:01.943788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.2.81.125:36455)
0924 12:52:01.945145 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.945149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.945167 (+ 18us) server_negotiation.cc:407] Connection header received
0924 12:52:01.945232 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.945236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.945388 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.945495 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:52:01.946181 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.947169 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.949322 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.950189 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.950503 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:01.951160 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:52:01.951836 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:52:01.951963 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:01.952300 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 12:52:01.952393 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1072,"threads_started":1}
I20260924 12:52:01.967043 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:01.959675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.2.81.124:46103)
0924 12:52:01.960440 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.960446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.960463 (+ 17us) server_negotiation.cc:407] Connection header received
0924 12:52:01.960561 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.960565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.960741 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.960898 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:52:01.961343 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.962371 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.964072 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.964765 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.965011 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:01.965655 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:52:01.966350 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:52:01.966500 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:01.966608 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 12:52:01.966702 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":515,"threads_started":1}
I20260924 12:52:01.984298 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:01.974840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.2.81.125:36455)
0924 12:52:01.975828 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.975835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.975976 (+ 141us) server_negotiation.cc:407] Connection header received
0924 12:52:01.976292 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.976298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.976387 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.976464 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:52:01.977404 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.978681 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.981297 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.982225 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.982371 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:01.983433 (+ 1062us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:52:01.983580 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:52:01.983640 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:01.983854 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 12:52:01.983946 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849}
I20260924 12:52:01.992568 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:01.976572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.2.81.124:46103)
0924 12:52:01.976757 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.976763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.986084 (+ 9321us) server_negotiation.cc:407] Connection header received
0924 12:52:01.986656 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.986662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.986766 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.986850 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:52:01.987550 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.988478 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.990058 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.990877 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.991038 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:01.991690 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:52:01.991841 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:52:01.991897 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:01.992177 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 12:52:01.992270 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 12:52:01.993676 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 12:52:01.975256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.2.81.126:39737)
0924 12:52:01.987049 (+ 11793us) server_negotiation.cc:206] Beginning negotiation
0924 12:52:01.987055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 12:52:01.987076 (+ 21us) server_negotiation.cc:407] Connection header received
0924 12:52:01.987219 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 12:52:01.987224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 12:52:01.987544 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 12:52:01.987867 (+ 323us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 12:52:01.988316 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.989416 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.991078 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 12:52:01.991489 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 12:52:01.991607 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 12:52:01.992109 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 12:52:01.992545 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 12:52:01.993294 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0924 12:52:01.993349 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 12:52:01.993427 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11548,"thread_start_us":11743,"threads_started":1}
I20260924 12:52:01.999034 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 25181
I20260924 12:52:02.000041 25243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:52:02.115747 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 25181
I20260924 12:52:02.142194 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 25255
I20260924 12:52:02.143342 25317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:52:02.258291 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 25255
I20260924 12:52:02.275132 2373 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskYQebFB/build/asan/bin/kudu pid 25330
I20260924 12:52:02.276327 25392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 12:52:02.384130 2373 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYQebFB/build/asan/bin/kudu with pid 25330
2026-09-24T12:52:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4956 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9350 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (372661 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST